In this tutorial, we’ll learn the basics of batch processing. Any time that you need to process a number of files using a repetitive pattern for the analysis, batch processing features in GemStone can save you time and provide consistency. The system is very flexible, allowing you to do different things to files in different groups. You can store your command scripts so that they can be reused the next time you need to repeat the process.
Background of the Example
The example we will use in this tutorial is a model for a research project to monitor fetal red blood cells in cases of fetomaternal hemorrhage (FMH). The model was developed in conjunction with Dr. Bruce H. Davis of Trillium Diagnostics, LLC. See “Multicenter Clinical Experience With Flow Cytometric Method for Fetomaternal Hemorrhage Detection”, Cytometry 50:285-290, 2002, published by Wiley-Liss, Inc for additional background.
The model uses three Cell Types, one for Adult RBCs, one for Adult F-Cells, and one for Fetal RBCs. We will set up batch processing for a set of files from different patients to quantify the relative percentages of each cell type.
First Time Processing
Open the GemStone Model Document
In most cases, you’ll use one GemStone model document for a given batch process, but it is certainly possible to use a set of model documents, each applied to a different group of files. We’ll keep things simple for this tutorial.
Click the Open Document button on the main toolbar. Navigate to the folder in which GemStone was installed and find the Model Examples folder. Select “FetalRBC.gs” and click Open.
Activate the Adult RBC Cell Type by selecting its tab at the bottom of the Cell Type workspace.
Batch Processing Tip: Open the GemStone Model Document First
It’s a good idea to open the GemStone document that you are going to use for batch processing as your first step. This allows you to inspect the model before you begin batch processing with it.
Select the Files to Analyze
Next, we select the FCS files that we want to analyze.
Click the Select FCS Files button on the main toolbar. Navigate to the folder in which GemStone was installed and find the Sample Files folder. Select the five “QQ” files in the list and click Open.
GemStone will add the files to the File Database panel and read the first file into our model.
Edit the Fields for the Database
Since this is the first time we’ve done this analysis, we’ll need to set up our database fields so that we store the information we want. Once we select the fields, we’ll save them to a disk file so that we do not have to repeat the process next time.
Right click the File Database panel and choose Edit Fields from the menu.
We want to define a new set of fields from scratch. In the Edit Columns dialog, click the Delete All button to remove any fields that might have been defined for another application. A few fields will be left in the list; these are fields that cannot be deleted.
Now we’re ready to select the new fields.
In the tree control on the left side of the dialog, you will see a section named Application. Within this section, find two fields: Application_Document Name and Application_Opened Document Name. Select these fields and click the Add button. These fields will store the name of the GemStone model document that we are using (FetalRBC.gs), as well as a GemStone document that we will save for each data file that we analyze.
Next, scroll down in the tree to the CellType1 section. In this group, select CellType1_Zone0_Adult RBC_NumberOfEvents and CellType1_Zone0_Adult RBC_PcntOfTotalEvents. Click the Add button to add these fields to our database.
Scroll to CellType2, and select two fields: CellType2_Zone0_Adult RBC_NumberOfEvents and CellType2_Zone0_Adult RBC_PcntOfTotalEvents. Click the Add button.
Finally, scroll to CellType3, and select two fields: CellType3_Zone0_Adult RBC_NumberOfEvents and CellType3_Zone0_Adult RBC_PcntOfTotalEvents. Click the Add button.
Our database is set up to store the number of events and percentages for each of the cell types in the model. Let’s save this database structure so that we can reuse it without having to find the results again in the tree.
Click the Save button on the right of the dialog. Navigate to a convenient location to store this file, type FetalRBCFields for the filename, and click Save.
Batch Processing Tip: Save your Database Structure
Since you will probably do more than one kind of analysis with GemStone, the information you want to store in a database will change. Use the Save and Open buttons in the Edit Columns dialog to store and retrieve the appropriate database structure for the analysis you are doing.
Click OK to close the Edit Columns dialog.
Set the Batch Options
Now that we have selected the files and set up our database, we’re ready to define what happens as each file is processed. For this, we use the Batch Options panel, found below the File Database panel on the left side of the GemStone workspace.
Here’s what we want the batch to do. After it loads one of our data files, we want it to re-open the FetalRBC model document. We do this so that the model is in the same state when we begin our analysis. Then, we want GemStone to trigger the Auto Analyisis command. And finally, we want the program to save the finished analysis in a separate GemStone model document.
In the Batch Options panel, enable the Open Document command by checking it. Then, click the Edit… button for Open Document Options.
In the dialog that is displayed, we will specify the GemStone document that we want to open.
Set the Location Option to Use the specified Target File.
Click the Target File property, and in the Choose File dialog, select the FetalRBC.gs document and click Open. Click OK to close the document options dialog.
In the Analysis section of the Batch Options panel, enable the Auto Analyze Data option, and uncheck any other options.
In the Finish Up section, enable the Save Document option and uncheck other options. Then, click the Edit… button for Save Document Options.
In the dialog that is displayed, we will specify where we want GemStone to save our finished analysis and how to name it.
Set the Location Option to Use the data file name and the Target Path. This will name the GemStone document using the data file name that was analyzed.
Click the Target Path property, and in the Choose Folder dialog, choose a folder that you have write-access to and click OK. Click OK to close the document options dialog.
Let’s save this set of batch options so that we can use the same commands again.
Just below the Batch Options panel, there are Open and Save buttons. Click Save, navigate to a convenient location, and type FetalRBCBatchCommands for the file name. Click Save to store the commands.
Batch Processing Tip: Save your Batch Options
You can save time setting up for batch processing by saving your batch options to disk. The next time you perform the same analysis, open your saved batch options and you’re ready to go.
Process the Files
We’re all set up and ready to go. The best approach for batch processing is to do the first file, make sure everything is working as planned, and then to let the batch process run automatically through the rest of the files.
Click the Redo button on the batch toolbar at the bottom of the GemStone workspace.
GemStone will open the first data file in the batch and then run the commands we selected. It opens the FetalRBC.gs model document, performs an Auto Analysis, and then saves a new GemStone model document based on the name of the data file we analyzed. We also see the analysis results in the File Database.
To automatically advance through the rest of the files, click the Auto Advance button on the batch toolbar.
As GemStone processes each file, it repeats the batch options that we selected and adds the analysis results to the File Database. Notice that the Application_Document Name field shows the name of the document that was saved with the finished analysis, and the Application_Opened Document Name shows the name of the original model document. These fields are very useful for keeping track of how files were analyzed and where the final analysis is stored.
After the First Time
If we wanted to run this batch process again tomorrow, our task would be easier because we have saved the database fields and the batch commands. Our process for setting up would be simplified:
1. Open the GemStone model document.
2. Select the FCS files to analyze.
3. If the file database fields need to be set up, we use Edit Fields, and then Open the file we saved (FetalRBCFields.xml) with the database structure.
4. Use the Open command at the bottom of the Batch Options panel to read in our FetalRBCBatchCommands.xml file.
5. Process the first file by clicking the Redo button. Inspect to make sure everything is working OK.
6. Process the remaining files by clicking the Auto Advance button.
Using Levels
There are situations in batch processing where you need to treat some files differently than others in the batch. For example, suppose you have a GemStone model document that is designed to analyze 8 markers for a 4-tube panel. When the first file in the panel is read, you need GemStone to open the appropriate GemStone model document from disk and do an analysis. When the second and third files are read, you just want the program to analyze the data. And when the last file in the panel is read, you need to analyze the data, save the document, and print a report. Your analysis is not complete until you have read and analyzed all four tubes.
In a situation like this, you can use Batch Levels to perform different actions on certain files in the panel. Here’s how you would handle the example described above.
After selecting the files for the cases you are going to analyze, expand the Batch Levels panel. This displays some additional controls that we will use to create the levels in our batch.
Next, you would select the second and third file in the list, and click the Indent button to indent the files to Level 2.
Similarly, you would select the fourth file in the list, and click the Indent button twice to indent the file to Level 3. If you have more than one case, you would repeat the process for the additional cases.
Now that you have 3 “levels” of files in the file database, you can set the batch options for each level separately. Notice that the Batch Options panel now shows a tab for each of the three levels.
To set the options for the Level 1 files, click the Level 1 tab. In our example, we would set the options on this tab to Open Document and Auto Analyze Data.
On the Level 2 tab, we would set the options to simply Auto Analyze Data.
And on the Level 3 tab, we would choose Auto Analyze Data, Print Workspaces, and Save Document.
As the files are processed, GemStone looks to see which level in the outline the files are on and selects the appropriate commands for that level.
This feature provides great flexibility for processing files automatically.
Summary
Batch processing with GemStone simplifies the task of processing more than one file at a time. A batch process requires some initial setup to configure the database and the processing options you want to perform. Once the process is defined, files can be automatically analyzed and results stored.
The batch system allows more advanced processing through the use of outline “levels” in the file database. Files on different levels can be processed in different ways. Levels are useful for processing panels of files and other processes that require more than one analysis strategy for the files in the batch.