Welcome to the RIFT experimental environment
============================================

Please refer to [LICENSE.md] and [COPYRIGHT.md] files first.

Then, familiarity with [https://datatracker.ietf.org/doc/draft-przygienda-rift/]
is assumed. To summarize


   **RIFT** is a specialized, dynamic routing protocol for
    Clos and fat-tree network topologies.  The protocol 
    
 1. deals with automatic construction of fat-tree topologies 
    based on detection of links, 
 2. minimizes the amount of routing state held at each level,
 3. automatically prunes the topology distribution exchanges to a
    sufficient subset of links, 
 4. supports automatic disaggregation of prefixes on link and node 
    failures to prevent black-holing and suboptimal routing, 
 5. allows traffic steering and re-routing policies, 
 6. allows non-ECMP forwarding, 
 7. automatically rebalances traffic towards the spines based on 
    bandwidth available and ultimately 
 8. provides mechanisms to synchronize a limited key-value
    data-store that can be used after protocol convergence to e.g.
    bootstrap higher levels of functionality on nodes.

This environment allows to generate and run topologies of RIFT protocol
possibly sharded over multiple machines. Nodes are connected via means of local 
UDP sockets with point-to-point links communicating using matched up ports.

This implementation consists of multiple parts, **parts annotated with (--) are 
not included in the public release package**:

 1. scripts to generate the necessary YAML files with topology
    description in `.../topology/scripts`
 2. some precomputed topology YAML files in `.../topology`
 3. RIFT environment binaries reading the file to execute RIFT nodes instances
    using the topology YAML files in `.../bin/rift-environ`
 4. scripts executing pre-canned scenarios in `.../scripts`
 5. `.../schemas` contains all the thrift schemas and serialization keys/format
    to decode the communication on links  
 6. (--) REDIS database listener that can listen to optionally published state
    from the RIFT nodes and deserialize it in `.../bin/rift-redis-listener`
 7. (--) equivalent REDIS database reader
 8. (--) REDIS database FIB builder and "pinger" in `.../bin/rift-redis-fabric-route-lookup`
 8. (--) `.../schemas` contain additional REDIS data schemas published by RIFT
 9. (--) `.../schemas` contains additional schemas for services **RIFT** exposes 
   and can be used to configure it during runtime and retrieve its operational 
   state
10. `MANIFEST.enc` is encrypted and can be ignored 
11. `publisher_key.pub.pkcs8` is the public key of the published package, 
   not used right now

Installation
============

As next, please read and follow the [INSTALL.md] file to make sure your environment has
enough resources and you have correct tools and packages to run **RIFT** environment.

Where to go next?
=================

Running scripts in .../scripts is a good way to verify that **RIFT** runs properly
in your environment. The scripts show how to use the internal CLI or use the
statistics obtained to derive convergence properties.

Running the binaries with a '--help' option provides a description of
different possibilities to customize **RIFT** nodes during runtime, e.g.

    ./bin/rift-environ --help

(--) Redis Persistency and Analytics
====================================

Running **RIFT** binaries with `REDIS` persistency options and using the listener
to retrieve different statistics and analytics data in real time is a good
way to observe some timings, e.g. run in two different terminals following 
commands (`rift-redis-listener` takes system identifiers of nodes without reading any 
kind of topology YAML files and hence -N1 is equivalent to registering for core_1
in the simple `two.yaml` topology)

    ./bin/rift-redis-listener -v -N1 -R -B
    ./bin/rift-environ -r redis://127.0.0.1 -t topology/two.yaml -D

Alternately, the redis database can be scanned for objects of nodes without
listening to publisher channels during runtime but it may be much slower

    ./bin/rift-redis-reader -v -N1 -S

Ultimately 

    ./bin/rift-redis-fabric-route-lookup
    
allows to construct the FIBs from the published routes and next hops and 
validate the available paths through the fabric and their weights.

Redis persistency schemas (with packet formats) can all be found in the
.../schemas directory. Those can be used to monitor traffic on the UDP
links or develop deserializers for the Redis data to obtain Thrift objects
for further processing. Observe that we use our own version of 
serialization of the data structures since `JSON` has several shortcomings and 
the schema used allows for very light clients that can monitor for a value of 
a single field in data structures at very low overhead. 

Scripts
=======

As next step analyzing the content of .../topology/scripts shows how to generate
different CLOS topologies. This is pretty sophisticated so you are on
your own largely.

The YAML file format is documented but may be changing between releases.
Looking at the example files and scripts generating it, it is pretty easy 
to understand.

Ultimately, you can run **RIFT** binaries with a simple, internal CLI by doing 

    ./bin/rift-environ topology -u ...
    
and enable internal **RIFT** thrift services which will allow you to read 
operational state and configure the protocol at runtime (the servers are enabled
by according `YAML` clauses in the topology). Look further in scripted examples. 

You can find the envelope of this release and future work in [TODO.md] and 
[CHANGELOG.md].

Reporting Problems
==================

**RIFT** does not produce cores except in very extreme cases.
It reports every failure by extensive logging and sometimes backtraces on exit.
Please provide the topologies you were using together with invocations and
resulting output in case of problems. We will likely need more detailed
runs with debug logs depending on the case.

Logging
=======

Under normal circumstances only critical down to error level is logged, to
obtain information and debug level logging, use multiple -v switches when running.
You can restrict logging per nodes, per peers and per components, as an
example to see debug logs of SPF on node "core_1"

    ./bin/rift-environ -Cspf -Ncore_1 -c -vvv  topology topology/two.yaml

FAQ
===

Q: I see following errors

    thread '<unnamed>' panicked at 'could not initialize thread_rng: Too many open files ...'

A: The environment doesn't get enough file descriptors, something like `ulimit -n 4096` will
   resolve the problem. **RIFT** environ can shard over multiple servers to extend the
   limits.
   
Q: How do I observe behavior without Redis? 

A: Please use `-DD` option that documents all internal structures and statistics on 
   exit and the mini CLI commands supported.

Q: I see large numbers of retransmissions, flooding losses, slow convergence or 

    ERRO cannot send hello: ... message: "No buffer space available"

A: UDP may be overwhelmed by the traffic **RIFT** environment generates. Refer to INSTALL notes
   how it can be improved on certain platforms.

   In some environments locally installed firewalls try to intercept all UDP communication and
   are overwhelmed quickly with the **RIFT** traffic. Switch off any local firewalling on the 
   machine.

Q: I see 

    CRIT cannot join multicast group 224.0.1.150
    
   and **RIFT** does not converge

A: You need to have networking enabled and adapters have forwarding entries on many
   platforms for multicast to function. Normally, making sure that you have an
   interface towards a default gateway will solve the problem.
   
Q: (--) How do I integrate **RIFT** into my environment 

A: For analytics you probably want to use the `Redis` capabilities and register for the according 
   information. You will either read very specific fields using the according keys or otherwise 
   have to write a de-serializer according to the schema documents. 
   
   If you just need to support your own `cli` the attached services schema and some generated 
   sample clients allow to use your language of choice to query and configure **RIFT** in an 
   asynchronous fashion. 
   
Q: I see 

    CRIT another RIFT process running ... 
    
   and **RIFT** does not start

A: As indicated, another **RIFT** process is already running and consumes resources which cannot 
   be shared with another instance. 
   
Q: I sometimes see

    WARN cannot determine level for node 1, reflecting node S-TIE
    
A: This can be safely ignored, a minor clause is not implemented yet leading to those warnings. 
   **RIFT** works correctly nevertheless. 
    
Q: I sometimes see on shutdown 

   CRIT Waiting for Idles expired, hard node shutdown with 4 peers not responding to Idle ...
   
A: This seems to happen under heavy load on few cores 4.* Linux kernels, the issue is cosmetic 
   and under investigation currently. 