Skip to content

Commit

Permalink
fix: more file -> bin renames
Browse files Browse the repository at this point in the history
  • Loading branch information
c-dilks committed Sep 17, 2024
1 parent 924a678 commit cff6f51
Show file tree
Hide file tree
Showing 8 changed files with 87 additions and 80 deletions.
107 changes: 57 additions & 50 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ The following tables describe the available datasets in the QADB. The columns ar
- _Up-to-Date_: this is the most recent Pass of these data, and the QADB has been updated for it
- _Deprecated_: a newer Pass exists for these data, but the QADB for this version is still preserved
- _TO DO_: the Pass for these data exist, but the QADB has not yet been updated for it
- **Data files**: the DST files used for the QA
- **Data files**: the input data files used for the QA

### Run Group A

Expand Down Expand Up @@ -80,30 +80,34 @@ The following tables describe the available datasets in the QADB. The columns ar

## Defect Bit Definitions

* QA information is stored for each DST file, in the form of "defect bits"
* QA information is stored for "QA bin", in the form of "defect bits"
* the user needs only the run number and event number to query the QADB
* queries will find the DST file associated with the event, and are only
* queries will find the QA bin associated with the event, and are only
performed "as needed"
* full dumps of the QADB are also possible, for browsing
* A QA bin is:
* the set of events between a fixed number of scaler readouts (roughly a time bin, although
there are fluctuations in a bin's duration)
* for older QADBs, Run Groups A, B, K, and M of Pass 1 data, the QA bins were DST 5-files
* N/F is defined as the electron yield N, normalized by the Faraday Cup charge F; the
electron yield is for Forward Detector electrons with `status<0`, unless specified otherwise
* The QA checks for outliers of N/F, along with several other miscellaneous criteria
* The term "golden" means that a file has ***no*** defects
* The table below lists the defect bits
* The term "golden" means that a QA bin has ***no*** defects
* The table below lists the defect bits
(Example: `defect=0b11000` has defects `SectorLoss` and `LowLiveTime`)

| Bit | Name | Description |
| --- | --- | --- |
| 0 | `TotalOutlier` | outlier N/F, but not terminal, marginal, or sector loss, for FD electron |
| 1 | `TerminalOutlier` | outlier N/F of first or last file of run, not marginal, for FD electron |
| 1 | `TerminalOutlier` | outlier N/F of first or last QA bin of run, not marginal, for FD electron |
| 2 | `MarginalOutlier` | marginal outlier N/F, within one standard deviation of cut line, for FD electron |
| 3 | `SectorLoss` | N/F diminished within a FD sector for several consecutive files |
| 3 | `SectorLoss` | N/F diminished within a FD sector for several consecutive QA bins |
| 4 | `LowLiveTime` | live time < 0.9 |
| 5 | `Misc` | miscellaneous defect, documented as comment |
| 6 | `TotalOutlierFT` | outlier N/F, but not terminal, marginal, or `LossFT`, FT electron |
| 7 | `TerminalOutlierFT` | outlier N/F of first or last file of run, not marginal, FT electron |
| 7 | `TerminalOutlierFT` | outlier N/F of first or last QA bin of run, not marginal, FT electron |
| 8 | `MarginalOutlierFT` | marginal outlier N/F, within one standard deviation of cut line, FT electron |
| 9 | `LossFT` | N/F diminished within FT for several consecutive files |
| 9 | `LossFT` | N/F diminished within FT for several consecutive QA bins |
| 10 | `BSAWrong` | Beam Spin Asymmetry is the wrong sign |
| 11 | `BSAUnknown` | Beam Spin Asymmetry is unknown, likely because of low statistics |
| 12 | `TSAWrong` | __[not yet used]__ Target Spin Asymmetry is the wrong sign |
Expand Down Expand Up @@ -131,18 +135,20 @@ The following tables describe the available datasets in the QADB. The columns ar
database format; future development plans include considering more
efficient formats, such as `SQLlite`
* there are also some text files stored in `text/`:
* **NOTE**: since both `OkForAsymmetry` and `Golden` criteria sets have been deprecated,
these text files will no longer be maintained; they are still provided, however
* `text/listOfGoldenRuns.txt`: list of runs, each classified as one of the following:
* `gold`: all files have no defects. Note that this is **very strict**,
so not many runs are `gold`, since most runs have at least one file
with a defect; in practice it is better to apply QA cuts per file,
* `gold`: all QA bins have no defects. Note that this is **very strict**,
so not many runs are `gold`, since most runs have at least one QA bin
with a defect; in practice it is better to apply QA cuts per QA bin,
using the QADB software
* `silver`: the only defects are terminal outliers (first or last file is
* `silver`: the only defects are terminal outliers (first or last QA bin is
an outlier); note that `gold` runs are, by definition, also `silver`.
This is also **very strict**: so far, only about half the runs are
`silver`
* `defect`: not `gold` or `silver`
* `text/listOfGoldenFiles.txt`: list of files with no defects
* `text/summary.txt`: summary table, where for each file the QA criteria result
* `text/listOfGoldenFiles.txt`: list of QA bins with no defects
* `text/summary.txt`: summary table, where for each QA bin the QA criteria result
(`Golden`, `OkForAsymmetry`, etc.) is provided, but note that these QA criteria
are deprecated, in favor users choosing their own criteria
* it is also possible to produce `latex` tables; see
Expand Down Expand Up @@ -178,9 +184,9 @@ Classes in both C++ and Groovy are provided, for access to the QADB within analy

## Table files
Human-readable format of QA result, stored in `qadb/*/*/qaTree.json.table`
* each run begins with the keyword `RUN:`; lines below are for each of that
run's file and its QA result, with the following syntax:
* `run number` `file number` `defect bits` `comment`
* each run begins with the keyword `RUN:`; lines below are for each of that
run's QA bins and its QA result, with the following syntax:
* `run number` `bin number` `defect bits` `comment`
* the `defect bits` are listed by name, and the numbers in the `[brackets]`
indicate which sectors have that defect
* if a comment is included, it will be printed after the defect bits, following the
Expand All @@ -194,21 +200,21 @@ Human-readable format of QA result, stored in `qadb/*/*/qaTree.json.table`
* the format is a tree (nested maps):
```
qaTree.json ─┬─ run number 1
├─ run number 2 ─┬─ file number 1
│ ├─ file number 2
│ ├─ file number 3 ─┬─ evnumMin
│ │ ├─ evnumMax
│ │ ├─ sectorDefects
│ │ ├─ defect
│ │ └─ comment
│ ├─ file number 4
│ └─ file number 5
├─ run number 2 ─┬─ bin number 1
│ ├─ bin number 2
│ ├─ bin number 3 ─┬─ evnumMin
│ │ ├─ evnumMax
│ │ ├─ sectorDefects
│ │ ├─ defect
│ │ └─ comment
│ ├─ bin number 4
│ └─ bin number 5
├─ run number 3
└─ run number 4
```
* for each file, the following variables are defined:
* for each bin, the following variables are defined:
* `evnumMin` and `evnumMax` represent the range of event numbers associated
to this file; use this to map a particular event number to a file number
to this bin; use this to map a particular event number to a bin number
* `sectorDefects` is a map with sector number keys paired with lists of associated
defect bits
* `defect` is a decimal representation of the `OR` of each sector's defect bits, for
Expand All @@ -220,47 +226,48 @@ qaTree.json ─┬─ run number 1
a similar format:
```
chargeTree.json ─┬─ run number 1
├─ run number 2 ─┬─ file number 1
│ ├─ file number 2
│ ├─ file number 3 ─┬─ fcChargeMin
│ │ ├─ fcChargeMax
│ │ ├─ ufcChargeMin
│ │ ├─ ufcChargeMax
│ │ └─ nElec ─┬─ sector 1
│ │ ├─ sector 2
│ │ ├─ sector 3
│ │ ├─ sector 4
│ │ ├─ sector 5
│ │ └─ sector 6
│ ├─ file number 4
│ └─ file number 5
├─ run number 2 ─┬─ bin number 1
│ ├─ bin number 2
│ ├─ bin number 3 ─┬─ fcChargeMin
│ │ ├─ fcChargeMax
│ │ ├─ ufcChargeMin
│ │ ├─ ufcChargeMax
│ │ └─ nElec ─┬─ sector 1
│ │ ├─ sector 2
│ │ ├─ sector 3
│ │ ├─ sector 4
│ │ ├─ sector 5
│ │ └─ sector 6
│ ├─ bin number 4
│ └─ bin number 5
├─ run number 3
└─ run number 4
```
* for each file, the following variables are defined:
* for each bin, the following variables are defined:
* `fcChargeMin` and `fcChargeMax` represent the minimum and maximum DAQ-gated
Faraday cup charge, in nC
* `ufcChargeMin` and `ufcChargeMax` represent the minimum and maximum FC charge,
but not gated by the DAQ
* the difference between the maximum and minimum charge is the accumulated charge
in that file
in that bin
* `nElec` lists the number of electrons from each sector


<a name="charge"></a>
# Faraday Cup Charge Access
* the charge is stored in the QADB for each DST file, so that it is possible to
* the charge is stored in the QADB for each QA bin, so that it is possible to
determine the amount of accumulated charge for data that satisfy your
specified QA criteria.
* see `src/examples/chargeSum.groovy` or `srcC/examples/chargeSum.cpp` for
usage example in an analysis event loop
* call `QADB::AccumulateCharge()` within your event loop, after your QA cuts
are satisfied; the QADB instance will keep track of the accumulated charge
you analyzed (accumulation performed per DST file)
you analyzed (accumulation performed per QA bin)
* at the end of your event loop, the total accumulated charge you analyzed is
given by `QADB::getAccumulatedCharge()`
* note: for Pass 1 QA results, we find some evidence that the charge from file to file may slightly overlap,
or there may be gaps in the accumulated charge between each file; the former leads to
* note: for Pass 1 QA results for Run Groups A, B, K, and M, we find some
evidence that the charge from bin to bin may slightly overlap,
or there may be gaps in the accumulated charge between each bin; the former leads to
a slight over-counting and the latter leads to a slight under-counting
* for RGK, we find the correction to this issue would be very small
(no more than the order of 0.1%)
Expand All @@ -286,7 +293,7 @@ Documentation for QADB maintenance and revision
* run `source environ.sh`
* run `bin/makeTables.sh`
* run `bin/makeTextFiles.sh`
* ~~update customized QA criteria sets, such as `OkForAsymmetry`~~
* ~~update customized QA criteria sets, such as `OkForAsymmetry`~~ this function is no longer maintained
* update the above table of data sets
* use `git status` and `git diff` to review changes, then add and commit to
git, and push to the remote branch
Expand Down
4 changes: 2 additions & 2 deletions src/examples/chargeSum.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ while(reader.hasEvent()) {

// accumulate charge; note that although the call to
// QADB::accumulateCharge() charge happens for each
// event within a DST file that passed the QA cuts, that
// file's charge will only be accumulated once, so
// event within a QA bin that passed the QA cuts, that
// bin's charge will only be accumulated once, so
// overcounting is not possible
qa.accumulateCharge()

Expand Down
2 changes: 1 addition & 1 deletion src/examples/cutCustom.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ QADB qa = new QADB()

// custom QA cut definition
// - decide which defects you want to check for; an event will not pass
// the QA cut if the associated file has any of the specified defects
// the QA cut if the associated bin has any of the specified defects
// - set to true to check the bit
// - set to false to ignore the bit (by default, all bits are ignored)
qa.checkForDefect('TotalOutlier',false)
Expand Down
22 changes: 11 additions & 11 deletions src/examples/dumpQADB.groovy
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// dump QADB data for a specified run; this demonstrates access to the various
// data stored in QADB
// - this program does not require a HIPO file or clas12root; it only loops
// through the QADB itself, performing lookup by file number
// through the QADB itself, performing lookup by bin number
// - you can specify a run number as an argument

// imports
Expand All @@ -23,26 +23,26 @@ QADB qa = new QADB()
//QADB qa = new QADB(5000,5500);


// loop through files
// loop through QA bins
int evnum
for(int filenum=0; filenum<=qa.getMaxBinnum(runnum); filenum++) {
for(int binnum=0; binnum<=qa.getMaxBinnum(runnum); binnum++) {
// skip non-existent bin numbers (required since old QADBs' bin numbers are multiples of 5)
if(!qa.hasBinnum(runnum, filenum)) { continue }
if(!qa.hasBinnum(runnum, binnum)) { continue }
sep("=",50)
println "FILE NUMBER $filenum"
println "BIN NUMBER $binnum"

// perform the lookup, by filenum
if(qa.queryByBinnum(runnum,filenum)) {
// perform the lookup, by binnum
if(qa.queryByBinnum(runnum,binnum)) {

// we need an event number within this file, to pass to QA criteria
// we need an event number within this QA bin, to pass to QA criteria
// checking methods, such as Golden; no additional Query will be called
evnum = qa.getEvnumMin();

// print whether this file passes some QA cuts
// print whether this bin passes some QA cuts
if(qa.golden(runnum,evnum)) {
println "- GOLDEN FILE!"
println "- GOLDEN BIN!"
} else {
println "- not golden: file has defects"
println "- not golden: bin has defects"
print (qa.OkForAsymmetry(runnum,evnum) ? "- OK" : "- NOT OK")
println " for asymmetry analysis"
}
Expand Down
4 changes: 2 additions & 2 deletions srcC/examples/chargeSum.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,8 @@ int main(int argc, char** argv) {

// accumulate charge; note that although the call to
// QADB::accumulateCharge() charge happens for each
// event within a DST file that passed the QA cuts, that
// file's charge will only be accumulated once, so
// event within a QA bin that passed the QA cuts, that
// bin's charge will only be accumulated once, so
// overcounting is not possible
qa->AccumulateCharge();

Expand Down
2 changes: 1 addition & 1 deletion srcC/examples/cutCustom.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ int main(int argc, char** argv) {

// custom QA cut definition
// - decide which defects you want to check for; an event will not pass
// the QA cut if the associated file has any of the specified defects
// the QA cut if the associated QA bin has any of the specified defects
// - set to true to check the bit
// - set to false to ignore the bit (by default, all bits are ignored)
qa->CheckForDefect("TotalOutlier",false);
Expand Down
4 changes: 2 additions & 2 deletions srcC/examples/cutGolden.cpp
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
// demonstrates how to select events from golden files only, which
// are files that have no defect
// demonstrates how to select events from golden bins only, which
// are bins that have no defect
// - you must specify a hipo file as an argument

#include <iostream>
Expand Down
22 changes: 11 additions & 11 deletions srcC/examples/dumpQADB.cpp
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// dump QADB data for a specified run; this demonstrates access to the various
// data stored in QADB
// - this program does not require a HIPO file or clas12root; it only loops
// through the QADB itself, performing lookup by file number
// through the QADB itself, performing lookup by bin number
// - you can specify a run number as an argument

#include <iostream>
Expand Down Expand Up @@ -31,26 +31,26 @@ int main(int argc, char ** argv) {
//QADB * qa = new QADB(5000,5500);


// loop through files
// loop through QA bins
int evnum;
for(int filenum=0; filenum<=qa->GetMaxBinnum(runnum); filenum++) {
for(int binnum=0; binnum<=qa->GetMaxBinnum(runnum); binnum++) {
// skip non-existent bin numbers (required since old QADBs' bin numbers are multiples of 5)
if(!qa->HasBinnum(runnum, filenum)) continue;
if(!qa->HasBinnum(runnum, binnum)) continue;
sep("=",50);
cout << "FILE NUMBER " << filenum << endl;
cout << "BIN NUMBER " << binnum << endl;

// perform the lookup, by filenum
if(qa->QueryByBinnum(runnum,filenum)) {
// perform the lookup, by binnum
if(qa->QueryByBinnum(runnum,binnum)) {

// we need an event number within this file, to pass to QA criteria
// we need an event number within this bin, to pass to QA criteria
// checking methods, such as Golden; no additional Query will be called
evnum = qa->GetEvnumMin();

// print whether this file passes some QA cuts
// print whether this bin passes some QA cuts
if(qa->Golden(runnum,evnum)) {
cout << "- GOLDEN FILE!" << endl;
cout << "- GOLDEN BIN!" << endl;
} else {
cout << "- not golden: file has defects" << endl;
cout << "- not golden: bin has defects" << endl;
cout << (qa->OkForAsymmetry(runnum,evnum) ? "- OK" : "- NOT OK");
cout << " for asymmetry analysis" << endl;
};
Expand Down

0 comments on commit cff6f51

Please sign in to comment.