Write Bulk Data From All Topics to a File

Estimated reading time: 3 minutes

This page describes how to write data from all Topics in OT Link Platform to a file. The flow will trim the data so that we only write data to the file that we are interested in. It will also prevent writing duplicate entries in the file by checking current values with previous values. Trimming the data and preventing duplicates ensures good performance when sending the data written to the file to a database or external storage. We will write the data to the /ftp-data/ directory on the OT Link Platform device so we can later send the data to a database, as described in the Send Bulk Data from a file to a Database page.

Prerequisites:

  • This flow uses a flow context level global variable. See Contexts and Globals for further information about global variables in OT Link Platform Flows.

Add a Flow to Create and Write to a File

Tip:

Try these two options to reduce the amount of data that we collect from data topics:

  1. devicehub.raw.> - This will send all data from DeviceHub tags only and no other sources of tags.
  2. devicehub.raw.{PLC_ID}.> - This will send all data from a specific PLC only.

To create a flow that writes all data to a file to send to a database:

  1. Connect a DataHub subscribe node, a Function node, and a File node for writing.

  2. Double-click the DataHub Subscribe node and enter > for the topic to subscribe to all topics.

  3. Double-click the Function node. Enter the following lines of code:

    var temp = (JSON.parse(msg.payload)).success; // Stores the boolean for if the message succeeded.
    var temp1 = (JSON.parse(msg.payload)).tagName; // Stores the tag name from the message.
    var d1  = new Date(); // Gets the current date and time.
    if(temp && temp1) // Checks if the message succeeded and if the tag name exists before continuing.
    {
        var obj = {}; // Creates a blank object that we will populate with 3 values.
        var datafilter = JSON.parse(msg.payload); // Makes an object out of the JSON in the message.
        obj.timestamp = datafilter.timestamp; // Puts the message timestamp from the JSON into our new object.
        obj.tagName = datafilter.tagName; // Puts the tagName from the JSON into our new object.
        obj.value = datafilter.value; // Puts the value from the JSON into our new object.
    
        var previous = flow.get(obj.tagName); // Gets the last tagname we wrote to the file.
        if(previous !== obj.value) // Ensures we only continue writing to the file if the current tagname is different than the last tagname.
        {
          msg.payload = JSON.stringify(obj); // Turns our object into a JSON string that we can send as a message.
          flow.set(obj.tagName,obj.value); // Stores the current tagname within the current flow so we can check it each time through this function.
    
    	 /* Ensures that our JSON message will write to a location that we have permission to write to in OT Link Platform 
         * called /ftp-data/. Our filename will be the current date and time. */
          msg.filename = "/ftp-data/"+d1.getMonth()+"-"+d1.getDate()+"-"+d1.getHours()+"-"+d1.getMinutes()+ ".txt";
          return msg; // Putting the return statement here ensures that the flow will only continue if the tagname changed.
        }
    }
    
  4. Click Done. Click Save

Whats Next?

We can use this flow for the final goal of writing the bulk data from the file to an external database. To continue with further steps, see Send Bulk Data from a file to a Database.

The final flow will look something like the below screenshot: