Flush

Flush

Sometimes a node is expected to finalize any ongoing operations, clear its buffers etc. immediately. A flush call is sent to all the nodes in these situations.

Flush is not a frequently used method when the client has connected to Data Flow for streaming, using websocket or grpc.

For batch projects where our clients send http requests, handling flush calls is a requirement for every node.

Flush Flow

When a flush has been received the nodes of that project are called one by one, starting from nodes with no input connections (ie. entry node etc.), and then the nodes that are connected to them recursively. This way when a node receives a flush call it is guaranteed that, all the nodes that provide input to this specific node has already completed their own flush handling.

A node is expected to focus on its own flush handling without expecting any more inputs from its connected input nodes.

Handling Flush

Trivial Case

For trivial nodes that keep no internal state and perform no async tasks (eg. Http requests), handling flush is trivial. Just do nothing. The default implementation will kick in and your node will be exempt from flush flow.

Nodes That Keep State

If your node keeps state or waiting for a flush call to finish its work etc. then you are expected to add a flush function to your node object. flush function does not have any parameters.
Your node is expected to call function onNodeFlushDone() when it is done with its own flush handling.

MyCustomNode.prototype.flush = function()
{
    //add your own flush handling code here
    //You can raise events, do logging etc.
    ...
    
    //call onNodeFlushDone() when the node has finished handling the flush
    onNodeFlushDone();
}

Asynchronous Case

There is no requirement to call onNodeFlushDone();inside the flush() function directly.

If your node keeps state and on top of that it performs some asynhronous tasks, like making http calls, then you should delay calling onNodeFlushDone(); until all your flush handling is done.

For example if you have one or more ongoing Http calls you should call onNodeFlushDone(); in one of those handlers, only after all of those response handlers are called. Also please beware of long running tasks. See details for flow of execution.

Remarks

In a batch request (ie. http) when all the nodes finish their flush handling the http response is generated with all the information gathered from the nodes so far.

There may be some async calls which does not effect the response that will be sent to the client. For example, if you send an http post request to some external server each time a flush is called, to count the number of flushes in some http server, then you do not need to wait for their completion since it is not an information that your client will observe in the output.