Column Processing Model

Column Processing

"Column processing" means the order in which columns are processed in each row of a Report and the order in which data and Expressions are processed.

Basically:  

• All data extraction from the database is done before any Expressions are processed.
• Columns are filled in in "left" (top of Columns screen) to "right" (bottom of Columns screen) order.
  • Data columns to the left of the current column are filled in and data columns to the right of the current column are empty/void/null.
  • If you are using a column as a filter for another column it must be to the left (above) the current column.
• "First Pass" Expressions are processed before the report is sorted, so the rows will be in the order in which they were pulled out of the database.
  • Expression columns are also processed left to right.  First pass Expression columns to the left are filled in and those the right are empty/null/void.
• Sorting is done after "first pass" Expressions are processed.
• "Second Pass" Expressions can count on all data extraction columns being filled in, all "first pass" Expression columns being filled in, and the rows being in sorted order.
• Sectioning and totals are done at the very end.
• The very last action is to prepare the output, based on the chosen "output format" for the report.

The ReportWriter works in "Phases":

Phases 1 to 3 - Data Extraction from the Database

In Phases 1 to 3 of a Report, data is pulled from the database column by column.  Columns on the left side of the report (and higher up in the list of columns on the 2. Columns screen) are extracted and filled before columns on the right side of the report (and lower down in the list of columns).  Earlier columns (higher up in the Report) are already filled and later columns (lower down in the report) do not yet exist.  Each row is completed before the next row is filled.

Output of Phase 1 & 2

The output of phase 1 is the set of rows extracted from the database:

All columns except the Expression columns are filled in with data.  The next step is to run the Expressions in the Expression columns.  This is Phase 2.

Phase 3 - Filtering the Data

In Phase 3, any filters (on the 4. Filters screen of the Report) that were not applied as part of the database queries in Phases 1 & 2 are now applied.  This removes all rows of the report that should not be part of the output.

Output of Phase 3

The output of Phase 3 is the final set of rows that will either be printed, exported (e.g. to Excel) or used to generate Forms.

Phase 4 - Expressions, Pass 1

Expressions are not 'run' until all of the data is already extracted from the database. There are two Expression processing passes, one before sorting and one after. That is, Expression Column processing takes place after all of the data is already in one big list of rows in memory.  Expressions can examine all existing data and it is possible to determine the current location in the matrix of data.   The first Expression pass takes place immediately after all data extraction from the database is complete.   All rows are examine again, columns processed in left to right order, and all Expressions marked for "first pass" processing are evaluated:

Expression Language Processing

First Pass Expression processing takes  place after all the data is brought into memory from the physical database, but before any sorting is done.   The Expression processor re-visits every row in the Report, visiting every Column in the row, from left to right (top  to bottom on the "Columns" screen of the report).   Expression  columns are filled in one by one.  These columns can "look behind" at any Expression column to the left of the current column (or in the previous row);  they cannot "look ahead" to Expression columns 'later' (to the right) of the current column or any Expression column  in the next row  (because the Expression processor has not visited those Columns as of yet).  

CAVEAT: Sorting (see below) is not done until after "first pass" Expression processing has finished:  you cannot depend on a sort for your "first pass" Expression processing.  Second pass Expression processing takes place after the sorting.   There are two Expression processing 'passes' over the report:  one before the rows are sorted, and a second one after the rows are sorted.

CAVEAT: Because "first pass" Expressions are processed prior to sorting, the first pass "rownumber" counter represents the row numbers in their "raw" unsorted state.  Once sorted, the row numbers may end up appearing in random order - they are not updated by the sort, they always retain their original (and now out of order) values.

CAVEAT: Any Expression column that contains either "prevRow" or "nextRow" should have the "Evaluate this expression after sorting rows" checkbox checked.   This warning is so important it also appears below.  Successful nextRow and prevRow processing is only guaranteed after the sorting is finished.

Phase 5 - Sorting the Rows and Document Sections and Totals

The rows in the Report are not sorted until after Expression processing (Phase 4) is completed.  So Expressions cannot rely on having sorted data.  Once the data is sorted, it is ready to be sectioned and printed (or, at least, output).

 After the data is sorted, if it is supposed to be broken into sections, the sections are created.  If there are no Document sections, this is skipped.

Finally, any Section and Report totals are computed.

Phase 5.5 - Expressions Pass 2,  Filtered and Sorting  (After Sort Expressions)  - WORKAROUND for Expressions Being Run Before Rows are Sorted.

If there are any expression columns that have the 'Evaluate this expression after sorting rows' checkbox checked, the second Expression pass is done and now all remaining Expression columns are evaluated.

CAVEAT: Any Expression column that contains either "prevRow" or "nextRow" should have the "Evaluate this expression after sorting rows" checkbox checked.

Phase 6 - Printing and Form Filling

The final phase of Report processing, the data is prepared for output.  The possibilities include:

• Generating an Adobe PDF of the "normal" column report.
• Exporting the rows of data to Excel
• Exporting the rows of data to a CSV, XML or Tab-separated text file.
• Printing Forms.  One PDF Form is created for each "section" of the Document/data.  A Document "Section" must exist for Forms to be generated (i.e. there must be a Section defined on the 6. Sections and Totals screen of the Report).   Multiple rows of data can be printed onto a single Form.  The "Document Section" indicates when to start the next Form.