Parquet File

Parquet files are used to store the data from log data files.

You can find more information about the Apache Parquet file format here.

A log data file will have one or more parquet files associated with it. For each CAN message listed in the log data file, there will be a corresponding parquet file.

Example;------------------------------------------------------------------------------- ; Message Number ; | Time Offset (ms) ; | | Bus ; | | | Type ; | | | | ID (hex) ; | | | | | Reserved ; | | | | | | Data Length Code ; | | | | | | | Data Bytes (hex) ... ; | | | | | | | | ; | | | | | | | | ;---+-- ------+------ +- --+-- ----+--- +- -+-- -+ -- -- -- -- -- -- -- 1) 0.000 1 Rx 0100 - 8 00 00 00 00 00 00 00 02 2) 0.000 1 Rx 0101 - 8 00 00 00 00 00 00 00 03 3) 1000.000 1 Rx 0101 - 8 00 00 00 00 00 00 00 04 4) 2000.000 1 Rx 0100 - 8 00 00 00 00 00 00 00 03 5) 2000.000 1 Rx 0101 - 8 00 00 00 00 00 00 00 01

In this example, CAN message MESSAGE_100 has hex ID 0100 and CAN message MESSAGE_101 has hex ID 0101.

For this log file, there would be two parquet files:

Parquet Columns

The parquet files contain the following columns:

  1. mid - Corresponds to the "Message Number" column in the log file.
  2. tm - The timestamp at which the message was recorded.
  3. bus - Corresponds to the "Bus" column in the log file.
  4. type - Corresponds to the "Type" column in the log file.
  5. Remaining columns - One for each CAN signal in the CAN message.
    • The values stored in the parquet files are not the actual values. They must have the CAN factor and offset applied in order to calculate the actual values.
Download Parquet Files

Users with at least OEM access can download the parquet files for a log data file. This is useful for debugging the raw data associated with a data file.

  1. Go to the data files page.
  2. Select a log data file.
  3. Click the button in the preview overlay.

The resulting ZIP file will contain the parquet files associated with the given data file. If there were any errors accessing any of the parquet files, their file names will be listed in a missing-parquets.txt file in the ZIP file.