Step Solutions - NXTStep

Release notes

v1.03

🚀 New Features

  • Publisher modes functionality in publisher
  • Data validator and backfiller components
  • Data gap creator for QA testing
  • Integration tests for backfilling QA
  • Commander GET endpoint

🐛 Fixes

  • Changed exporter dependencies to standard utilities
  • Added missing compression to commander messages
  • Subscriber topic lookup now handles wildcards correctly
  • Updated Pydantic model to support new API config table format
  • Resolved connector duplicate configuration issue
  • Cleaned up asset registration in OPC UA interface
  • Resolved collector issue that was discarding high-frequency measurements
  • Fixed commander backward compatibility issue
  • Resolved commander serialization issues
  • Fixed deadlock during MQTT disconnect
  • Reduced number of database reconnects in connector
  • Resolved deprecated database configuration for data validator
  • Corrected incorrect type-casting in command executor
  • Resolved race condition during database disconnect in command executor
  • Fixed Pydantic model issues in command executor
  • Corrected status handling for validation errors in command executor
  • Resolved reference issue in command executor

🛠 Improvements

  • Added cycle time configuration to access layer
  • Enhanced query and system tests for connector
  • Automatic code linting report
  • Updated versions for external dependencies

⚠️ Deprecations

  • Deprecated arguments for internal logger

v1.02.01

🐛 Fixes

  • Hotfix MQTT socket timeout on network loss

v1.02

🚀 New Features

  • New access layer integrated into collector and commander
  • Added health monitoring and message encryption in MQTT client
  • Support for multiple MQTT protocols (paho.mqtt v2)
  • OPC UA improvements: datatype lookup, support for node path lists
  • Expanded Pydantic models across all core modules
  • Clarify integration in exporter

🐛 Fixes

  • Improved reliability of MQTT (reconnects, race conditions)
  • Fixed OPC UA errors with missing nodes and configuration issues
  • Corrected collector and commander configuration handling
  • Fixed issues with BACnet boolean writes and executor field naming
  • Resolved multiple integration test and API issues

🛠 Improvements

  • More efficient OPC UA node lookups
  • Enhanced logging and error reporting across modules
  • Cleaner configuration and dependency management

v1.01.02

🚀 New Features

  • Internal features

🐛 Fixes

  • Internal fixes to improve security and stability
  • Bumped dependency versions after CVE revision

🛠 Improvements

  • Improved OPC-UA lookup
  • Internal improvements to security

v1.0

First release of new NXTStep core.

NXTStep demo environment

Introduction

This is the documentation for the Step Solutions NXTStep demo API. We’ve configured a simulated DSS environment, where we host some of the interfaces we provide.

The data is generated by a series of simulated instances of data streams, representing an environment of several edge computers streaming data to our NXTStep DSS.

Intended use and data representation

The data from the API is first and foremost intended for graphing purposes. As a result, the data might have gone through some processes on the way to your endpoint, to ensure that the returned data is of an appropriate amount to quickly and efficiently serve data for a graphing frontend. The data might therefore not be exactly the raw measured value, but a representation of it.

There is a /user_profile endpoint in the API which returns a set of parameters for your specific API instance. Consult the Swagger UI for details.

The returned data can be divided into these categories:

  • Truncated data
  • Data extracted from raw data source (e.g. 1 second measurement interval)
  • Data extracted from lower resolution data source (e.g. 1 minute/hour/day measurement interval)
Truncated data

Data of a different resolution than the source you’re extracting the data from. Normally used when we want to transfer a smaller amount of data than what is contained in the data source within that time interval. For example, if you have raw data measured every 1 seconds (60 measurements per minute), and you want this truncated as “1 minute data”, the 60 measured points are combined, and represented as one data point. In this process, the resulting data point might be altered to best represent the removed data.

Data extracted from raw data source

The measurement data streamed from the site(s), is stored as raw time series data.

Data extracted from lower resolution data source

In addition to storing the time series data as raw data, it is periodically copied to tables with a lower resolution. We transfer a truncated representation of the raw data, according to the intended interval of the table it’s copied to. Using the same example as before, if the raw table has measurements every 1 seconds, and it’s copied into a “1 minute” table, the minute table has 1/60 as many datapoints as the raw table. A “1 hour” table has 1/3600 of the raw data table.

API return values

With the information in the previous paragraph, we can explain how the data is returned from the API.

There are two variables that define what data is returned:

  • Raw or lower resolution
  • Unmodified or truncated

Whether the data is raw data, or from a lower resolution source depends on how long the time frame of your request is. This is parametrized on an instance basis, depending on assigned processing resources. Default raw data range is time interval < 24 h.

Whether the data is truncated or not, also depends on the requests time frame. This is also parametrized on an instance basis configured maximum samples. The default maximum samples is 500. If you have a 1 second measurement interval, the requests time frame has to be below 500 seconds in order to get raw data. However, when calculating the resolution of truncated data, the number of seconds between each sample is truncated itself, not rounded. It’s therefore possible that the returned amount is above 500. If the time interval of the request is 15 minutes, which is 900 seconds, the requested number of seconds between each sample is 900/500 = 1,8 seconds, but 1,8 seconds truncated is 1 second, so the returned number of samples is 900.

If your service agreement has a “raw data” pricing point enabled, the API allows you to bypass the truncating, by setting the “aggregate_function” parameter in your request to “off”. The time frame for the request is still limited depending on the amount of sampled data that is stored, to prevent unprocessably large data sets. This is only activated and allowed when your service agreement allows for it. This ensures that you don’t get truncated data within the raw data interval, and allows for bigger time intervals of raw data.

Examples

Using data sources with 1 second measurement interval

1 second time interval = 60 samples/minute = 3600 samples/hour.

Example 1

Time interval: 5 minutes = 300 seconds

Potential number of returned samples: 300

Result: Time interval < 24 h. Raw data table is the source. Time interval < 500 seconds, unmodified data is returned. 300 samples.

Example 2

Time interval: 10 minutes = 600 seconds

Potential number of returned samples: 600

Result: Time interval < 24 h. Raw data table is the source. Time interval > 500 seconds, truncated data is returned, every 600/500 = 1,2 (1) seconds. 600 samples.

Example 3

Time interval: 1 hour = 60 minutes = 3600 seconds

Potential number of returned samples: 3600

Result: Time interval < 24 h. Raw data table is the source. Time interval > 500 seconds, truncated data is returned, every 3600/500 = 7,2 (7) seconds. ~514 samples.

Example 4

Time interval: 12 hours = 720 minutes = 43200 seconds

Potential number of returned samples: 43200

Result: Time interval < 24 h. Raw data table is the source. Time interval > 500 seconds, truncated data is returned, every 43200/500 = 86,4 (86) seconds. ~502 samples.

As these examples show: When requesting time series data from the API, the data might be truncated in order to fit the maximum number of requested samples.

Aggregate function

The API endpoints include a parameter (aggregate_function) so you can specify what kind of data altering you want to happen in the truncating process:

  • Lowest value of the total points in the truncated interval (min)
  • Highest value of the total points in the truncated interval (max)
  • The value of the first point in the truncated interval (first)
  • The value of the last point in the truncated interval (last)
  • The average value of the values in the truncated interval (avg) (default)
  • Only raw data (off). Only available when contractually stipulated.

However, this parameter does not take into account if the truncated data is extracted from the raw data or the lower resolution source. You are not able to control how the data in the low resolution table is stored, only how it’s manipulated if truncating is done to decrease the number of return points. If the time interval of your requested data is high enough for the data to be extracted from a lower resolution source, the data is already truncated.

NOTE 1: It’s worth mentioning that this averaging is not true time averaging, but a “change-of-value” calculation. For example, if you have 1 minute of data measured every 10 seconds, and the value stays the same, e.g. 0, for the first 30 seconds, then is increased on the 40 and 50 second measurement point, you have 3 “points of interest”: 0, 40, 50 seconds. These are then summarized and divided by 3, as opposed to “true time averaging”, where all 6 would be summarized and divided by 6.

The conclusion of this chapter is: Tread lightly if you want to use the return values for raw data analysis. Get in touch with your Step Solutions representative for help ensuring raw data.

Additional restrictions

In order to ensuer a stable communication, we have additional restrictions on the data requests:

  • Maximum 25 datapoints (nxtstep_id) per request. If you try to go beyond that, only the first 25 will be used.
  • Maximum 5 requests per second. This is not a firm maximum, but if there are too many incoming requests, the API will have to restrict the responses to ensure stability, and this is dependent on the available resources. If this restriction happens, you will receive 429 responses (Too Many Requests). If you experience this, try queuing your requests to stay within the limit.
  • Per default, an access token has a lifetime of 1 hour. There is no need to acquire a new token until this lifetime is approaching its end. The token acquisition logs are analyzed, and a misused token endpoint will be deactivated without warning.

Client interaction

Swagger UI

If you want to try out our API, you can check out the interactive Swagger UI.

To authenticate, press the “Authorize” button on the right hand side of the page. The client ID field should be autofilled. The client secret field can be left blank, and the “scopes” selection can be left as is.

When you try to authenticate, you will be redirected to a login page, where you will be required to identify yourself with an approved M365 account. Get in touch with your Step Solutions representative for this access.

After you’ve been authenticated, you can check out the API endpoints. Consult the Swagger documentation for each endpoint to see its intended use.

Other methods

You can also test the API interaction with your prefered client representation (scripting, Postman, cURL etc.). For simplicity, the examples are cURL commands.

First, contact your Step Solutions representative to obtain the following:

  • Tenant ID
  • Client ID
  • Client secret

These will be used to acquire a token from our Oauth 2.0 server. This token can then be used in your API requests.

To acquire your token, a HTTP POST is sent to the Oauth server:

curl --location 'https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token' \
        --header 'Content-Type: application/x-www-form-urlencoded' \
        --data-urlencode 'grant_type=client_credentials' \
        --data-urlencode 'client_id=CLIENT_ID' \
        --data-urlencode 'scope=api://CLIENT_ID/.default' \
        --data-urlencode 'client_secret=CLIENT_SECRET'

The return will be of type JSON:

{
    "token_type": "Bearer",
    "expires_in": 3599,
    "ext_expires_in": 3599,
    "access_token": "token_is_here"
}

Use the expiration time to determine when you should acquire a new token. At the time of writing, the lifetime is 1 hour (3600 seconds), but this might be changed at a later stage.

After the token is acquired, this can be used in the header of the API request. The name of the header parameter is “Authorization” and the data should be “Bearer TOKEN”, i.e. “Authorization: Bearer TOKEN” (where TOKEN = the actual token from the Oauth response).

curl --location 'https://demo01.stepsolutions.no/api/time_interval?NXTStep_id=16_1_10016&search_time_span=day&amount=4&aggregate_function=last' \
--header 'Authorization: Bearer TOKEN'