Cloud Processing

Cloud Processing

What does the Cloud do?

The AQGateway Cloud processes the raw data recorded by the EDT and which was transferred to the cloud via the AQGateway App.
Once data has been uploaded to the Cloud, it is validated and the calibration information is applied to the data. Four bespoke algorithms are then applied to transform the data from raw data to compensated data.
A summary overview of the data can be accessed by selecting Time Plots. This has the advantage of knitting together all the data recorded by an EDT. The timescale over which data is to be viewed can be changed as required.
Detailed information for each batch is available for download. The data that is downloaded includes the raw data and the compensated data. This ensures that there is a full data audit trail.

What are the algorithms used to convert raw data to compensated data?

4 bespoke algorithms are applied to the raw data. In the Settings YAML file, one of these 4 algorithms, is identified as a suggested one to use and a second is identified as an alternative suggested . However, as the outcome of all 4 algorithms is made available in the downloaded data, so the choice of which algorithm to use for further processing remains with the user.

All algorithms subtract the WE offset from the measured WE counts. The first three algorithms use the AE current as a zero offset with offset temperature correction. The temperature dependent coefficients were determined at Alphasense in environmental chambers. You may find that if you observe consistent bias errors then the coefficients for these algorithms can be improved; this bias is most likely due to the environment where the EDT is being used. Contact Alphasense to discuss.

Algorithm 1 corrects the WE by subtracting the temperature corrected AE.

Algorithm 2 corrects the WE by subtracting the temperature corrected AE, scaled by the ratio of the WE and AE sensitivities.

Algorithm 3 corrects the WE by subtracting the temperature corrected AE, then further subtracting the difference in the offsets between the WE and AE.

Algorithm 4 ignores the AE and corrects the WE by subtracting a temperature dependent generic offset. This will normally need an offset adjustment for best accuracy. Algorithm 4 works with both 3 electrode and 4 electrode electrochemical gas sensors.

I've downloaded my processed files. What does it mean?

After downloading and opening the zip folder, you will see the following files :-

NAME File Contents
settings.EDT_nnnnnnnnn.yaml about the data file settings

log.EDT_nnnnnnnnn_01Jan20_120102.txt status of data processing stage

EDT_nnnnnnnnn_01Jan20_120102.yaml information about the data file

EDT_nnnnnnnnn_01Jan20_120102.hex raw data file from the EDT

compensated. EDT_nnnnnnnnn_01Jan20_120102.csv compensated EDT data

raw. EDT_nnnnnnnnn_01Jan20_120102.csv raw (pre-processed) EDT data

If you open the file starting with compensated. EDT_nnnnnnnnn_01Jan20_120102.csv - this displays the compensated AQ data. Separate the data on the comma's and the data is as follows :-

Column title                Description
timestamp               Day/mo/yr_hr:min:sec
record_no               Begins with 1, consecutive increasing
temp                    °C temperature 0.1 °C res’n (°F not available)
rh                      Relative Humidity 0.1 % resolution
we_count                Working Electrode A/D counts
ae_count                Auxiliary Electrode A/D counts
we_corrected            WE count with corrected offset subtracted
Algo_recommended        Recommended algorithm (1 to 4)
algo_used               Actual Algorithm used (1 to 4)
compensated_ppb     Compensated ppb using actual algorithm
algo_alternative            Second best algorithm (1-4)
we_corrected_alternative    WE count with corrected offset subtracted
compensated_alter_ppb   Compensated ppb using altern. algorithm
ppb_1                   Compensated ppb using algorithm 1
ppb_2                   Compensated ppb using algorithm 2
ppb_3                   Compensated ppb using algorithm 3
ppb_4                   Compensated ppb using algorithm 4

How do I find out why my batch says Failed?

Click on the batch that has failed to get the Batch Details page.
Click on the View button against the Run that has failed to get the File Processing Details page.

At the bottom of the page is a Report.
The most common reason for a batch to fail is that there is no data. In this case, the following message is displayed:

No EDT file found to process

There were no readings file included in this batch or the readings file was incorrectly named. The system is expecting a file that begins EDT2_ and ends .hex. If this batch was created by a gateway device, then there may have been an error while the data was being downloaded from the EDT.

To check if this is the case, Click the Results Download Link and download the .zip file. Look to see if the readings file beginning EDT2_ and ending .hex is in the zip file.

The temperature dependence does not look right. What should I do?

The temperature dependent coefficients used in processing the data were determined at Alphasense in environmental chambers. You may find that if you observe consistent bias errors then the coefficients for these algorithms can be improved; this bias is most likely due to the environment where the EDT is being used. Contact Alphasense to discuss.

How can I have negative PPB measurements?

Do not worry if some of the measurements show negative concentrations which are not possible.
Within the cloud, an offset is applied to the measurements recorded by the EDT. This offset was determined by placing the sensor in a zero-air chamber.
If your results are showing negative concentrations, this reflects the local environment and it is recommended that you calibrate your EDT to your local environment and to update the offset to reflect this.

I have a csv file with air quality data. Can I process it with AQGateway?

Currently we can only processes csv files of data from a specified EDT as we require information on the individual sensor in order to process data accurately.

The format of the csv file must contain at least the following columns with example data shown.

2019-12-27 10:49:00,0,10,-37,26,64.2
2019-12-27 10:49:00,1,6.4,-36,27,64.2
2019-12-27 10:50:00,2,6.4,-35,28,63.7
2019-12-27 10:50:00,3,6.4,-36,27,63.7
2019-12-27 10:51:00,4,6.4,-37,27,64.3
2019-12-27 10:51:00,5,6.4,-35,27,64.3

How do I setup an Admin user who can view and change the backend of AqGateway?

A user can only be assigned admin rights by a superuser. Admin rights are granted by checking the Staff Status checkbox and my adding the group Alphasense Admins to the groups this user belongs to.

How can I change the EDT Calibration values?

The EDT's are calibrated before dispatch. If a change of calibration (also referred to as resetting the zero baseline) is required, please see the section on how to calibrate EDT's for the methodology to identify the appropriate WE and AE values.

In order to change the EDT calibration, log onto the AQGateway.
Select My EDTs from the menu options on the left of the Home Screen.
Select the EDT for which the calibration is to be changed
On the EDT Info screen, there is a section on the bottom right of the screen entitled Update Baseline Calibrations.
Change the value for WE0 and AE0 to the required value
These values will be applied to all future processing. To apply this calibration to historic batches, select the relevant batch and reprocess the data.

What is Reference data and how do I upload it?

Reference data comes from devices located in the same place as EDTs are can be uploaded to compare with EDT readings. Currently only NO2 data can be uploaded.

The expected format is a csv and must have headers and include the columns:

Column format Further Info
timestamp yyyy-mm-dd hh:mm:ss
temp float -20 to 40 - 1dp precision
rh float 0 to 100 - 1dp precision
no2_ppb integer 0 to 300

Data that is deemed invalid is excluded from the processed csv file.

Reference data is uploaded to graphana and can be included on the same chart as EDT data.

Why do I need to reprocess a batch?

All EDT's are set up with an offset which is established in zero air at Alphasense
When using an EDT to measure air quality in your own environment, it may be appropriate to update this offset to reflect a more appropriate baseline
Data already recorded and uploaded to the Cloud can be reprocessed and the revised offset will be applied to the data.

Contact a human