top of page

iMonnit CSV Reconciliation: Remove Duplicate Messages Without Losing Real Readings

19 minutes ago
4 min read

Two CSV exports overlap. After joining them, a technician sees repeated temperatures and clicks Remove Duplicates on the value column. The resulting table is shorter, but it may have lost genuine observations. Repeated values are normal when conditions remain steady; a repeated export row is a different problem.


For iMonnit records, reconcile message identity first, then investigate missing periods. This guide proposes a controlled spreadsheet workflow for maintenance and monitoring teams. It does not change sensor calibration, reporting intervals or the original records.


Original concept illustration of overlapping sensor-data exports flowing into a checked archive.
Original explanatory illustration of export reconciliation. Generic hardware and data sheets; not an iMonnit screen or customer installation.

Key takeaways

Identify messages before removing rows. Equal measurements can be separate valid observations. Retain the source files and reconcile coverage separately from duplicates.


1 Keep identity separate from the measurement


Monnit’s CSV field definitions identify MessageID as the database message identifier and SensorID as the way to distinguish sensors, even when their names match. Retain both columns. A user-assigned name is useful context, but it is a poor sole key for merging records.


The same guidance distinguishes transformed Value, portal-style Formatted Value and Raw Data. Choose the documented field appropriate to the analysis and retain the others for reconciliation. Do not mix raw and transformed numbers in one temperature series or assume formatted text is directly suitable for arithmetic.


2 Preserve the files before combining them


Keep an untouched copy of each download. In a separate working folder, record the source account, export method, selected sensor or network, requested dates and download time. Confirm the time convention from the actual export and account settings rather than labelling a date column UTC by assumption.


  1. Import each file into a working table using explicit column types. Preserve identifiers as text and inspect the imported dates before sorting.

  2. Add a source-file column. This records where each row came from after the tables are combined.

  3. Check that both files use the same column meanings, units and intended scope. Split incompatible datasets before merging.

  4. Append the rows without deleting anything, then retain this combined starting table as an intermediate result.


Do not let a spreadsheet’s automatic formatting become an undocumented conversion. Check a few imported identifiers and timestamp strings against the original CSV. If the software cannot preserve a field reliably, stop and use an appropriate import method.


3 Reconcile repeated messages, not equal values


Group candidate duplicates by the documented message identifier, retaining SensorID as a cross-check. Compare the other relevant fields within each group. Identical message identities and matching contents across overlapping files can represent the same exported record.


If matching identifiers carry different values, times or sensor identities, retain the conflicting rows and investigate the account, export transformation and import process. Do not silently choose the newest file. If an export lacks reliable identifiers, record that limitation and agree a method with the data owner before removing rows.


Hypothetical export reconciliation: five input rows contain four unique messages; M101 and M102 both measure 4.2 degrees Celsius and must both remain.
Worked example using fictional message IDs. Only the matching M103 record overlaps; equal values from M101 and M102 remain separate observations.

In the hypothetical example, Export A contains M101, M102 and M103; Export B contains M103 and M104. Five input rows become four unique messages after confirming that both M103 rows match. M101 and M102 both report 4.2 °C, but deleting one would remove a genuine observation.


Keep an exception sheet containing every removed duplicate, its retained counterpart and source files. Record the starting count, matching duplicates removed, conflicts retained and final count. Someone else should be able to reproduce the result without relying on memory.


4 Prove coverage independently


A duplicate-free table can still be incomplete. Compare the first and last usable records, sensor coverage and observed timestamp gaps against the requested period. Do not manufacture missing rows by interpolation and present them as measured values.


Monnit’s historical reporting guidance states that the Sensor Overview download has a hard limit of 5,000 records. It separately documents Sensor Export: By Date Range for a single sensor, with up to 70,000 records from the past year. These are different retrieval routes; verify the available options in the actual account.


If a download reaches a limit or misses the required boundary, choose an appropriate supported retrieval route and reconcile the resulting files again. An overlapping date window helps compare shared records, but does not prove all readings exist. Exporting more files cannot recover observations that were never retained.


5 Close the handover with evidence


  • Retain originals, the working method, final dataset and exception sheet together.

  • Document the analysis field, units, time convention, sensor identities and any unresolved conflicts.

  • Have the record owner review unexplained gaps before using the dataset for a quality or operational decision.


For wider system planning, see planning an industrial data logging system. Browse the ProSense Monnit range when checking the installed sensor and gateway family. Account features and export behaviour must be verified for the actual software edition; this guide does not promise identical behaviour in iMonnit Express or other platforms.


Need help with your application?


ProSense Instruments - discuss instrumentation and data-recording requirements. Email sales@prosenseinstruments.com.au or call 1800 560 854.


ProSense Solutions - discuss a complete monitoring or automation application: sales@prosense.com.au.


ProSight Software - discuss software requirements: info@prosightsoftware.com. Confirm any device integration separately; this article does not establish direct Monnit compatibility.

Comments


bottom of page