OAP (PMS-2D) File format
Starting in 2007, with the PACDEX project, Optical Array Probe data files will have the following XML based header at the start of the file. The data records will follow and are the same as they have been all along, with the exception that tas is no longer encoded.
<?xml version="1.0" encoding="ISO-8859-1"?> <OAP version="1"> <Source>ncar.ucar.edu</Source> <FormatURL>http://www.eol.ucar.edu/raf/Software/OAPfiles.html</FormatURL> <Project>PLOWS</Project> <Platform>C130_N130AR</Platform> <FlightNumber>ff02</FlightNumber> <FlightDate>11/01/2009</FlightDate> <probe id="C4" type="Fast2DC" resolution="25" nDiodes="64" clockFreq="12" laserWaveLength="658" serialnumber="F2DC003" suffix="_LPO"/> <probe id="C6" type="Fast2DC_v2" resolution="10" nDiodes="64" clockFreq="33.333" laserWaveLength="660" serialnumber="F2DC002" suffix="_LPC"/> <probe id="P1" type="TwoDP" resolution="200" nDiodes="32" serialnumber="2DP10" suffix="_RWI"/> </OAP>
Additional meta-data may present. Each line is ended with a new-line character ('\n').
PMS-2D Record Formats
The PMS-2D raw data comes in 4 kilo-byte records of 32 bit slices from the probe (1024 slices). The interface card then wraps a record header around that with time stamp and other pertinent information. The Fast2D raw data comes in 4 kilo-byte records of 64 bit slices from the probe (512 slices).
The data is stored in big-endian format, so all data will need to be swapped when used on Intel architecture (Linux or MS Windows).
OAP Logical Record Format
/* Values currently in use for the 'id' field. */
#define PMS2D_C1 0x4331 // First PMS-2DC
#define PMS2D_C2 0x4332 // Second PMS-2DC
#define PMS2D_C4 0x4334 // RAF 64 diode 25 um Fast 2DC
#define PMS2D_C6 0x4336 // RAF 64 diode 10 um Fast 2DC
#define PMS2D_G1 0x4731 // First 2D Greyscale; unused to date
#define PMS2D_H1 0x4831 // SPEC HVPS
#define PMS2D_P1 0x5031 // First PMS-2DP
#define PMS2D_P2 0x5032 // Second PMS-2DP
struct P2d_rec {
short id; /* 'P1','C1','P2','C2', H1, etc */
short hour;
short minute;
short second;
short year; /* starting in 2007 w/ PACDEX */
short month; /* starting in 2007 w/ PACDEX */
short day; /* starting in 2007 w/ PACDEX */
short tas; /* true air speed */
short msec; /* msec of this record */
short overld; /* overload time, msec */
unsigned char data[4096]; /* image buffer */
};
typedef struct P2d_rec P2d_rec;
- id
- The id word tells whether it is a 2d-C, 2d-P or an HVPS. The 1 or 2 signifies first or second probe. Hex values are 'C1' = 0x4331, 'P1' = 0x5031 etc.
- Time stamp
- The hour, minute, second, and msec belong to the LAST slice of the record. This time stamp is stamped the moment the VME interface card is interrupted by the probe when a record is ready for downloading (ADS2) or the after the data has been transfered from the probe to the USB driver (ADS3) in the data acquisition system.
- True Air Speed
- The true air speed (tas) is packed in the same format as shipped to the probe. If the XML primary file tag is <PMS2D> then decode tas as follows: tas_float = rec.tas * 125 / 255. If the tag is <OAP> then there is no decoding, the tas is a truncated integer.
- Overload
- The overld value is the amount of time in milliseconds the probe was shut off while the interface card was unloading data. For ADS2 data (~1996 - ~2005) the overld value belongs to the NEXT logical data record (at least for the VME card used since May 1997, SHEBA).
For ADS2 data, typical values when in heavy cloud should be on the order of 37 milliseconds for 1 probe attached to the VME interface card and 50-60 milliseconds if 2 probes are attached. The probe can typically fill the buffer in as fast as 3 milliseconds.
For ADS3 this value will always be 0 and the overload word in the data buffer signifies an overload in the data. A difference between that and the previous timing word will give you the dead time.
PMS-2D Raw Data Format
Data are recorded in 32 bit slices, which are inverted (data are 0's and blanks are 1's). Starts with a "sync" word, 0xff000000, followed by N particle slices, followed by 3 blank slices, followed by a "timing" word.
- Sync-word
- The manual defines the sync word as 0x55000000, however, in an attempt to move to a 32 bit timing word the sync word changed to 0xff000000. RAF probes also produce 3 blank slices, compared to 1 as defined in the manual. As of DYCOMS-II (mid-2001), RAF has put the sync word back to 0x55000000.
- Timing slice
- The timing slice begins with 0x55, then the following 24 bits are the number of true air speed clock pulses which have elapsed since the last blank slice was recorded.
From a PMS-2D manual:
Bit
0 0011111111111111111111100
1 1111111111111111111111111
2 0011111111111111111111100
3 1111111111111111111111111
4 0011111111111111111111100
5 1111111111111111111111111
6 0011111111111111111111100
7 1111111111111111111111111
8 0011111111101111111111100
9 0011111111000111111111100
10 0011111110000011111111100
11 0011111100000001111111100
12 0011111000000000111111100
13 0011110000000000011111100
14 1011100000000000001111100
15 1011000000000000000111110
16 0010000000000000000011100
17 0000000000000000000001100
18 1000000000000000000000110
19 0010000000000000000000110
20 0011000000000000000001100
21 0011100000000000000011110
22 1011110000000000000111100
23 1011111000000000001111110
24 0011111100000000011111100
25 0011111110000000111111100
26 0011111111000001111111110
27 1011111111100011111111110
28 0011111111110111111111100
29 1011111111111111111111110
30 0011111111111111111111100
31 1011111111111111111111100
^ ^^^
| ||| Sync word
| Sync word. ||
|| Timing word.
|
| All '1's, end of particle.
A code snippet I use for locating the beggining of a particle:
if (slice == 0x55000000 && pSlice != 0xffffffff && ppSlice == 0xffffffff)
StartOfParticle();
where slice is the current slice, pSlice is the previous slice, and ppSlice is 2 slices previous.
A code snippet I use for getting the time of a particle:
frequency = probe.resolution / tas; timeWord = slice & 0x00ffffff; deltaTime = (unsigned long)((float)timeWord * frequency); *** Note: The 24 bit timing word is typically rolled over in about 8 seconds: the deltaTime is unknown if next particle is detected 8 seconds later.
PMS-2D Processed Data Format
The nimbus processor can produce 1D histograms from the 2D-C and 2D-P probe data. These are in the standard PMS-1D format as described in the NCAR-RAF netCDF conventions.
RAF Fast-2D Raw Data Format
Data are recorded in 64 bit slices and inverted: data are 0's and blanks are 1's. A particle, consisting of N data slices, is terminated by the sync/timing word. Blank slices may appear before or after the particle. There are two versions of this probe. The original was used from 2006 through 2018 (PACDEX - SOCRATES). Starting in August of 2018 a new electronics board was developed to handle a 10 micron version. We attmpeted to maintain as much backwards compatibility as possible.
Version 1:
- Sync word
- The new RAF Fast-2DC probe defines the sync word as 0xAAAAAA0000000000LL. It is in the first 3 bytes of the leading slice.
- If the sync word is 0xAAAAAB0000000000LL then the DOF flag has been triggered and the particle is outside the depth of field. Due to a bug in the firmware this flag is rarely set.
- If the Sync word is 0x5555AA000000000000LL, then data has been lost. This is the equivalent of the overld value used in the original 2D data. Subtracting this time word from the previous time word will give you the dead time (duration when probe was not sending data).
- Timing tag
- The time tag takes the last 5 bytes' space in the leading slice. This is a running 12 MHz clock starting from probe power on. The timing mask would be 0x000000ffffffffffLL
Version 2:
Sync word
The new RAF Fast-2DC probe defines the sync word as 0xAAAA000000000000LL. It is in the first 2 and a half bytes of the leading slice.
If the sync word is 0xAAAA100000000000LL then the DOF flag has been triggered and the particle is outside the depth of field. This is effective in this version of the probe.
If the Sync word is 0x555500000000000000LL, then data has been lost. This is the equivalent of the overld value used in the original 2D data. Subtracting this time word from the previous time word will give you the dead time (duration when probe was not sending data).
Timing tag
The time tag takes the last 5+ bytes space in the leading slice. This is a running 33 MHz clock starting from probe power on. The timing mask would be 0x000003ffffffffffLL
For both versions of the probe it is effective to just check the first two bytes for 0xAAAA or 0x5555.
A Fast-2DC particle structure:
Bit
0 11111111111111111111111
1 01111111111111111111110
2 11111111111111111111111
3 01111111111111111111110
4 11111111111111111111111
5 01111111111111111111110
6 11111111111111111111111
7 01111111111111111111110
8 11111111111011111111111
9 01111111110001111111110
10 11111111100000111111111
11 01111111000000011111110
12 11111110000000001111111
13 01111100000000000111110
14 11111000000000000011111
15 01110000000000000001110
16 11100000000000000000111
17 01000000000000000000010
18 10000000000000000000001
19 01100000000000000000000
20 11110000000000000000011
21 01111000000000000000110
22 01111100000000000001110
23 01111110000000000011110
24 01111111000000000111110
25 01111111100000001111110
26 01111111110000011111111
27 11111111111000111111111
28 01111111111101111111110
29 11111111111111111111110
30 11111111111111111111111
31 11111111111111111111110
32 11111111111111111111111
33 01111111111111111111110
34 01111111111111111111111
35 11111111111111111111111
..
.
64
^
|
| Sync/timing word for this particle.
A code snippet I use for locating the beginning of a particle:
if ((slice & 0xFFFF000000000000LL) == 0xAAAA000000000000LL)
StartOfParticle();
if ((slice & 0x0000010000000000LL) == 0x0000010000000000LL) // First version of probe.
RejectThisParticleDOF();
if ((slice & 0xFFFF000000000000LL) == 0x5555000000000000LL) // Overload
AddToDeadTime();
A code snippet I use for getting the time of a particle:
timeTag = slice & 0x000000FFFFFFFFFFLL; // Or 0x00003fffffffffffLL time = timeTag/12000 (in milli-seconds) // Or 33000 for the newer version