Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement OODT Crawler for injecting staged products into File Manager #8

Closed
lewismc opened this issue Feb 27, 2018 · 4 comments
Closed

Implement OODT Crawler for injecting staged products into File Manager #8

lewismc opened this issue Feb 27, 2018 · 4 comments
Assignees
@lewismc
Labels
crawler OODT Crawler Component enhancement filemgr OODT File Manager Service help wanted
Milestone
0.1

Comments

@lewismc
Copy link
Member

lewismc commented Feb 27, 2018

Once we have implemented #7 we need to look at implementing OODT's Crawler component for injecting staged products into the File Manager.
@lewismc lewismc added this to the 0.1 milestone Feb 27, 2018
@lewismc lewismc added crawler OODT Crawler Component filemgr OODT File Manager Service labels Sep 14, 2018
@lewismc
Copy link
Member Author

lewismc commented Sep 28, 2018

This can be done as follows

$ cd coal-sds-deploy/crawler/bin
$ ./bin/crawler_launcher --filemgrUrl http://localhost:9000 --operation --launchMetCrawler --clientTransferer org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory --productPath /usr/local/coal-sds-deploy/data/staging --metExtractor org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor --metExtractorConfig /usr/local/coal-sds-deploy/data/met/tika.conf

The output is as follows

Bad level value for property: org.apache.oodt.cas.crawl.typedetection.level
Bad level value for property: org.apache.oodt.cas.crawl.config.level
Bad level value for property: org.apache.oodt.cas.crawl.level
Bad level value for property: org.apache.oodt.cas.crawl.action.level
Bad level value for property: org.apache.oodt.cas.crawl.util.level
Setting property 'StdProductCrawler.filemgrUrl'
Setting property 'MetExtractorProductCrawler.filemgrUrl'
Setting property 'AutoDetectProductCrawler.filemgrUrl'
Setting property 'MetExtractorProductCrawler.metExtractor'
Setting property 'MetExtractorProductCrawler.metExtractorConfig'
Setting property 'StdProductCrawler.clientTransferer'
Setting property 'MetExtractorProductCrawler.clientTransferer'
Setting property 'AutoDetectProductCrawler.clientTransferer'
Setting property 'StdProductCrawler.productPath'
Setting property 'MetExtractorProductCrawler.productPath'
Setting property 'AutoDetectProductCrawler.productPath'
Sep 28, 2018 2:31:31 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey
FINE: Property 'StdProductCrawler.productPath' set to value [/usr/local/coal-sds-deploy/data/staging]
Sep 28, 2018 2:31:31 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey
FINE: Property 'MetExtractorProductCrawler.metExtractorConfig' set to value [/usr/local/coal-sds-deploy/data/met/tika.conf]
Sep 28, 2018 2:31:31 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey
FINE: Property 'MetExtractorProductCrawler.clientTransferer' set to value [org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory]
Sep 28, 2018 2:31:31 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey
FINE: Property 'AutoDetectProductCrawler.filemgrUrl' set to value [http://localhost:9000]
Sep 28, 2018 2:31:31 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey
FINE: Property 'AutoDetectProductCrawler.clientTransferer' set to value [org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory]
Sep 28, 2018 2:31:31 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey
FINE: Property 'StdProductCrawler.filemgrUrl' set to value [http://localhost:9000]
Sep 28, 2018 2:31:31 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey
FINE: Property 'AutoDetectProductCrawler.productPath' set to value [/usr/local/coal-sds-deploy/data/staging]
Sep 28, 2018 2:31:31 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey
FINE: Property 'MetExtractorProductCrawler.metExtractor' set to value [org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor]
Sep 28, 2018 2:31:31 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey
FINE: Property 'MetExtractorProductCrawler.filemgrUrl' set to value [http://localhost:9000]
Sep 28, 2018 2:31:31 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey
FINE: Property 'StdProductCrawler.clientTransferer' set to value [org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory]
Sep 28, 2018 2:31:31 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey
FINE: Property 'MetExtractorProductCrawler.productPath' set to value [/usr/local/coal-sds-deploy/data/staging]
Sep 28, 2018 2:31:31 PM org.apache.oodt.cas.crawl.ProductCrawler crawl
INFO: Crawling /usr/local/coal-sds-deploy/data/staging
Sep 28, 2018 2:31:31 PM org.apache.oodt.cas.crawl.ProductCrawler handleFile
INFO: Handling file /usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr
Sep 28, 2018 2:31:31 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Invoking tika extractor on file [/usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Number of captured tika metadata keys: [4]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Added tika met key [X-Parsed-By] with value [org.apache.tika.parser.DefaultParser]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Added tika met key [Content-Encoding] with value [ISO-8859-1]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Added tika met key [content] with value [ENVI
description = {
  Georeferenced Image built from input GLT. [Tue Aug 11 15:39:46 2015] [Tue
  Aug 11 16:17:51 2015]}
samples = 1363
lines = 22826
bands = 432
header offset = 0
file type = ENVI Standard
data type = 4
interleave = bil
sensor type = Unknown
byte order = 0
map info = { UTM , 1.000 , 1.000 , 201603.137 , 4061363.983 , 2.7000000000e+00 , 2.7000000000e+00 , 13 , North , WGS-84 , units=Meters , rotation=90.00000000 }
wavelength units = Nanometers
correction factors = { 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.880586 , 1.741631 , 1.524632 , 1.564904 , 1.390035 , 1.342501 , 1.264768 , 1.200357 , 1.131471 , 1.06512 , 1.105211 , 1.081234 , 1.064999 , 1.056798 , 1.042573 , 1.009621 , 1.009031 , 1.002107 , 0.997332 , 0.976807 , 0.963755 , 0.969028 , 0.962823 , 0.949522 , 0.960568 , 0.957813 , 0.934766 , 0.944994 , 0.937726 , 0.935257 , 0.932706 , 0.932568 , 0.933217 , 0.928705 , 0.929294 , 0.936669 , 0.935498 , 0.94823 , 0.949846 , 0.945885 , 0.935468 , 0.930084 , 0.934473 , 0.935378 , 0.939193 , 0.935081 , 0.937398 , 0.943396 , 0.947133 , 0.950645 , 0.945531 , 0.940295 , 0.933129 , 0.930664 , 0.92736 , 0.931786 , 0.928217 , 0.928205 , 0.926481 , 0.928583 , 0.930504 , 0.93648 , 0.930731 , 0.931265 , 0.935063 , 0.93434 , 0.926983 , 0.932689 , 0.936477 , 0.939647 , 0.940155 , 0.937519 , 0.939448 , 0.942124 , 0.93653 , 0.9435 , 0.959204 , 0.942566 , 0.940873 , 0.939414 , 0.939822 , 0.940174 , 0.941372 , 0.939347 , 0.942108 , 0.942664 , 0.934811 , 0.934567 , 0.937712 , 0.940611 , 0.944809 , 0.939877 , 0.943376 , 0.939189 , 0.943619 , 0.946268 , 0.940166 , 0.953752 , 0.958975 , 0.954512 , 0.954103 , 0.958978 , 0.953247 , 0.952199 , 0.956082 , 0.957846 , 0.970078 , 0.973704 , 0.980014 , 0.928845 , 0.922973 , 0.954414 , 0.95521 , 0.961276 , 0.964513 , 0.965296 , 0.964644 , 0.954999 , 0.951133 , 0.956216 , 0.951977 , 0.948547 , 0.949499 , 0.952685 , 0.950158 , 0.944263 , 0.936946 , 0.938394 , 0.941325 , 0.94116 , 0.941397 , 0.940811 , 0.942695 , 0.945228 , 0.953929 , 0.962457 , 0.968728 , 0.963947 , 0.961222 , 0.963003 , 0.967658 , 0.969773 , 0.970294 , 0.963456 , 0.970497 , 0.976972 , 0.961611 , 0.953081 , 0.945668 , 0.993867 , 1.019915 , 0.997013 , 0.977643 , 0.998022 , 1.007041 , 1.003881 , 0.991335 , 0.976202 , 0.967636 , 0.969294 , 0.965331 , 0.968705 , 0.965705 , 0.973601 , 0.97282 , 0.970848 , 0.970687 , 0.969394 , 0.972263 , 0.969286 , 0.970327 , 0.97754 , 0.984703 , 0.993916 , 1.02186 , 1.054704 , 1.061183 , 1.036962 , 1.012519 , 0.991209 , 0.975974 , 0.965446 , 0.958801 , 0.951519 , 0.960628 , 0.957276 , 0.96061 , 0.95689 , 0.956666 , 0.965567 , 0.982251 , 1.038526 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.131643 , 1.150333 , 1.112466 , 1.06255 , 1.033182 , 1.03018 , 1.006512 , 0.993285 , 1.003557 , 1.014441 , 0.998586 , 0.994426 , 0.987975 , 0.984681 , 0.981863 , 0.964106 , 0.956432 , 0.954467 , 0.956429 , 0.950668 , 0.947217 , 0.944635 , 0.942942 , 0.941078 , 0.943502 , 0.950408 , 0.957625 , 0.965073 , 0.976012 , 0.976683 , 0.975625 , 0.968011 , 0.968843 , 0.970632 , 0.960977 , 0.960505 , 0.955015 , 0.953597 , 0.951119 , 0.945679 , 0.949988 , 0.951236 , 0.947813 , 0.948004 , 0.950015 , 0.939258 , 0.945863 , 0.953927 , 0.953145 , 0.945291 , 0.942319 , 0.947022 , 0.948264 , 0.947112 , 0.942092 , 0.943128 , 0.948068 , 0.944432 , 0.950396 , 0.964006 , 0.961019 , 0.951786 , 0.957457 , 0.950327 , 0.954375 , 0.9608 , 0.965864 , 0.982396 , 1.011334 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.289798 , 1.126375 , 1.074621 , 1.069365 , 1.051364 , 1.009432 , 0.984578 , 0.991187 , 1.016306 , 1.083681 , 1.076539 , 1.104657 , 1.114005 , 1.081102 , 1.039196 , 1.009524 , 0.966515 , 0.953688 , 0.966274 , 0.973232 , 0.966402 , 0.946801 , 0.951952 , 0.965679 , 0.96977 , 0.946541 , 0.941678 , 0.937528 , 0.922351 , 0.914192 , 0.92879 , 0.932284 , 0.933182 , 0.922322 , 0.91851 , 0.919591 , 0.925027 , 0.924611 , 0.932288 , 0.933352 , 0.930517 , 0.931666 , 0.931763 , 0.932655 , 0.928945 , 0.933308 , 0.932392 , 0.932943 , 0.935328 , 0.947019 , 0.954093 , 0.95156 , 0.939591 , 0.942808 , 0.944862 , 0.944004 , 0.949161 , 0.950992 , 0.956738 , 0.951184 , 0.953545 , 0.958836 , 0.966134 , 0.956752 , 0.951961 , 0.958667 , 0.9579 , 0.968531 , 0.973792 , 0.969238 , 0.970838 , 0.954552 , 0.968166 , 0.989176 , 0.974784 , 0.970674 , 0.9733 , 0.990576 , 1.0062 , 1.010295 , 0.99378 , 0.986109 , 1.007054 , 1.005377 , 1.010013 , 1.014671 , 1.021618 , 1.021229 , 1.021003 , 1.020866 , 1.029358 , 1.042136 , 1.030482 , 1.019556 , 1.036656 , 1.05348 , 1.015947 , 1.07263 , 1.092879 , 1.053624 , 1.086491 , 1.139334 , 1.163645 , 1.162487 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 }
wavelength = { 346.6325373 , 351.6412073 , 356.6498773 , 361.6585473 , 366.6672173 , 371.6758873 , 376.6845573 , 381.6932273 , 386.7018973 , 391.7105673 , 396.7192373 , 401.7279073 , 406.7365773 , 411.7452473 , 416.7539173 , 421.7625873 , 426.7712573 , 431.7799273 , 436.7885973 , 441.7972673 , 446.8059373 , 451.8146073 , 456.8232773 , 461.8319473 , 466.8406173 , 471.8492873 , 476.8579573 , 481.8666273 , 486.8752973 , 491.8839673 , 496.8926373 , 501.9013073 , 506.9099773 , 511.9186473 , 516.9273173 , 521.9359873 , 526.9446573 , 531.9533273 , 536.9619973 , 541.9706673 , 546.9793373 , 551.9880073 , 556.9966773 , 562.0053473 , 567.0140173 , 572.0226873 , 577.0313573 , 582.0400273 , 587.0486973 , 592.0573673 , 597.0660373 , 602.0747073 , 607.0833773 , 612.0920473 , 617.1007173 , 622.1093873 , 627.1180573 , 632.1267273 , 637.1353973 , 642.1440673 , 647.1527373 , 652.1614073 , 657.1700773 , 662.1787473 , 667.1874173 , 672.1960873 , 677.2047573 , 682.2134273 , 687.2220973 , 692.2307673 , 697.2394373 , 702.2481073 , 707.2567773 , 712.2654473 , 717.2741173 , 722.2827873 , 727.2914573 , 732.3001273 , 737.3087973 , 742.3174673 , 747.3261373 , 752.3348073 , 757.3434773 , 762.3521473 , 767.3608173 , 772.3694873 , 777.3781573 , 782.3868273 , 787.3954973 , 792.4041673 , 797.4128373 , 802.4215073 , 807.4301773 , 812.4388473 , 817.4475173 , 822.4561873 , 827.4648573 , 832.4735273 , 837.4821973 , 842.4908673 , 847.4995373 , 852.5082073 , 857.5168773 , 862.5255473 , 867.5342173 , 872.5428873 , 877.5515573 , 882.5602273 , 887.5688973 , 892.5775673 , 897.5862373 , 902.5949073 , 907.6035773 , 912.6122473 , 917.6209173 , 922.6295873 , 927.6382573 , 932.6469273 , 937.6555973 , 942.6642673 , 947.6729373 , 952.6816073 , 957.6902773 , 962.6989473 , 967.7076173 , 972.7162873 , 977.7249573 , 982.7336273 , 987.7422973 , 992.7509673 , 997.7596373 , 1002.7683073 , 1007.7769773 , 1012.7856473 , 1017.7943173 , 1022.8029873 , 1027.8116573 , 1032.8203273 , 1037.8289973 , 1042.8376673 , 1047.8463373 , 1052.8550073 , 1057.8636773 , 1062.8723473 , 1067.8810173 , 1072.8896873 , 1077.8983573 , 1082.9070373 , 1087.9157073 , 1092.9243673 , 1097.9330373 , 1102.9417073 , 1107.9503873 , 1112.9590473 , 1117.9677273 , 1122.9763873 , 1127.9850573 , 1132.9937273 , 1138.0024073 , 1143.0110673 , 1148.0197373 , 1153.0284073 , 1158.0370873 , 1163.0457473 , 1168.0544273 , 1173.0630973 , 1178.0717673 , 1183.0804373 , 1188.0891073 , 1193.0977773 , 1198.1064473 , 1203.1151173 , 1208.1237873 , 1213.1324573 , 1218.1411273 , 1223.1497973 , 1228.1584673 , 1233.1671373 , 1238.1758073 , 1243.1844773 , 1248.1931473 , 1253.2018173 , 1258.2104873 , 1263.2191573 , 1268.2278273 , 1273.2364973 , 1278.2451673 , 1283.2538373 , 1288.2625073 , 1293.2711773 , 1298.2798473 , 1303.2885173 , 1308.2971873 , 1313.3058573 , 1318.3145273 , 1323.3231973 , 1328.3318673 , 1333.3405373 , 1338.3492073 , 1343.3578773 , 1348.3665473 , 1353.3752173 , 1358.3838873 , 1363.3925573 , 1368.4012273 , 1373.4098973 , 1378.4185673 , 1383.4272373 , 1388.4359073 , 1393.4445773 , 1398.4532473 , 1403.4619173 , 1408.4705873 , 1413.4792573 , 1418.4879273 , 1423.4965973 , 1428.5052673 , 1433.5139373 , 1438.5226073 , 1443.5312773 , 1448.5399473 , 1453.5486173 , 1458.5572873 , 1463.5659573 , 1468.5746273 , 1473.5832973 , 1478.5919673 , 1483.6006373 , 1488.6093073 , 1493.6179773 , 1498.6266473 , 1503.6353173 , 1508.6439873 , 1513.6526573 , 1518.6613273 , 1523.6699973 , 1528.6786673 , 1533.6873373 , 1538.6960073 , 1543.7046773 , 1548.7133473 , 1553.7220173 , 1558.7306873 , 1563.7393573 , 1568.7480273 , 1573.7566973 , 1578.7653673 , 1583.7740373 , 1588.7827073 , 1593.7913773 , 1598.8000473 , 1603.8087173 , 1608.8173873 , 1613.8260573 , 1618.8347273 , 1623.8433973 , 1628.8520673 , 1633.8607373 , 1638.8694073 , 1643.8780773 , 1648.8867473 , 1653.8954173 , 1658.9040873 , 1663.9127573 , 1668.9214273 , 1673.9300973 , 1678.9387673 , 1683.9474373 , 1688.9561073 , 1693.9647773 , 1698.9734473 , 1703.9821173 , 1708.9907873 , 1713.9994573 , 1719.0081273 , 1724.0167973 , 1729.0254673 , 1734.0341373 , 1739.0428073 , 1744.0514773 , 1749.0601473 , 1754.0688173 , 1759.0774873 , 1764.0861573 , 1769.0948273 , 1774.1034973 , 1779.1121673 , 1784.1208373 , 1789.1295073 , 1794.1381773 , 1799.1468473 , 1804.1555173 , 1809.1641873 , 1814.1728573 , 1819.1815273 , 1824.1901973 , 1829.1988673 , 1834.2075373 , 1839.2162073 , 1844.2248773 , 1849.2335473 , 1854.2422173 , 1859.2508873 , 1864.2595573 , 1869.2682273 , 1874.2768973 , 1879.2855673 , 1884.2942373 , 1889.3029073 , 1894.3115773 , 1899.3202473 , 1904.3289173 , 1909.3375873 , 1914.3462573 , 1919.3549373 , 1924.3635973 , 1929.3722773 , 1934.3809373 , 1939.3896073 , 1944.3982873 , 1949.4069573 , 1954.4156173 , 1959.4242873 , 1964.4329673 , 1969.4416373 , 1974.4502973 , 1979.4589673 , 1984.4676373 , 1989.4763173 , 1994.4849773 , 1999.4936473 , 2004.5023273 , 2009.5109973 , 2014.5196673 , 2019.5283373 , 2024.5370073 , 2029.5456773 , 2034.5543473 , 2039.5630173 , 2044.5716873 , 2049.5803573 , 2054.5890273 , 2059.5976973 , 2064.6063673 , 2069.6150373 , 2074.6237073 , 2079.6323773 , 2084.6410473 , 2089.6497173 , 2094.6583873 , 2099.6670573 , 2104.6757273 , 2109.6843973 , 2114.6930673 , 2119.7017373 , 2124.7104073 , 2129.7190773 , 2134.7277473 , 2139.7364173 , 2144.7450873 , 2149.7537573 , 2154.7624273 , 2159.7710973 , 2164.7797673 , 2169.7884373 , 2174.7971073 , 2179.8057773 , 2184.8144473 , 2189.8231173 , 2194.8317873 , 2199.8404573 , 2204.8491273 , 2209.8577973 , 2214.8664673 , 2219.8751373 , 2224.8838073 , 2229.8924773 , 2234.9011473 , 2239.9098173 , 2244.9184873 , 2249.9271573 , 2254.9358273 , 2259.9444973 , 2264.9531673 , 2269.9618373 , 2274.9705073 , 2279.9791773 , 2284.9878473 , 2289.9965173 , 2295.0051873 , 2300.0138573 , 2305.0225273 , 2310.0311973 , 2315.0398673 , 2320.0485373 , 2325.0572073 , 2330.0658773 , 2335.0745473 , 2340.0832173 , 2345.0918873 , 2350.1005573 , 2355.1092273 , 2360.1178973 , 2365.1265673 , 2370.1352373 , 2375.1439073 , 2380.1525773 , 2385.1612473 , 2390.1699173 , 2395.1785873 , 2400.1872573 , 2405.1959273 , 2410.2045973 , 2415.2132673 , 2420.2219373 , 2425.2306073 , 2430.2392773 , 2435.2479473 , 2440.2566173 , 2445.2652873 , 2450.2739573 , 2455.2826273 , 2460.2912973 , 2465.2999673 , 2470.3086373 , 2475.3173073 , 2480.3259773 , 2485.3346473 , 2490.3433173 , 2495.3519873 , 2500.3606573 , 2505.3693273 }
fwhm = { 5.55165 , 5.55524 , 5.55879 , 5.5623 , 5.56577 , 5.5692 , 5.5726 , 5.57595 , 5.57927 , 5.58255 , 5.58579 , 5.58899 , 5.59216 , 5.59529 , 5.59839 , 5.60144 , 5.60446 , 5.60745 , 5.6104 , 5.61332 , 5.61619 , 5.61904 , 5.62185 , 5.62463 , 5.62737 , 5.63008 , 5.63275 , 5.6354 , 5.63801 , 5.64058 , 5.64313 , 5.64564 , 5.64812 , 5.65057 , 5.65299 , 5.65537 , 5.65773 , 5.66005 , 5.66235 , 5.66461 , 5.66684 , 5.66905 , 5.67122 , 5.67337 , 5.67549 , 5.67758 , 5.67964 , 5.68167 , 5.68367 , 5.68565 , 5.6876 , 5.68952 , 5.69141 , 5.69328 , 5.69512 , 5.69694 , 5.69873 , 5.70049 , 5.70223 , 5.70394 , 5.70563 , 5.70729 , 5.70893 , 5.71055 , 5.71214 , 5.7137 , 5.71524 , 5.71676 , 5.71826 , 5.71973 , 5.72118 , 5.72261 , 5.72401 , 5.72539 , 5.72676 , 5.72809 , 5.72941 , 5.73071 , 5.73198 , 5.73324 , 5.73447 , 5.73569 , 5.73688 , 5.73806 , 5.73921 , 5.74034 , 5.74146 , 5.74256 , 5.74363 , 5.74469 , 5.74573 , 5.74676 , 5.74776 , 5.74875 , 5.74972 , 5.75067 , 5.7516 , 5.75252 , 5.75342 , 5.75431 , 5.75518 , 5.75603 , 5.75687 , 5.75769 , 5.75849 , 5.75928 , 5.76006 , 5.76082 , 5.76156 , 5.76229 , 5.76301 , 5.76371 , 5.7644 , 5.76507 , 5.76573 , 5.76638 , 5.76702 , 5.76764 , 5.76825 , 5.76884 , 5.76943 , 5.77 , 5.77056 , 5.77111 , 5.77164 , 5.77217 , 5.77268 , 5.77319 , 5.77368 , 5.77416 , 5.77463 , 5.77509 , 5.77554 , 5.77598 , 5.77641 , 5.77683 , 5.77725 , 5.77765 , 5.77804 , 5.77843 , 5.7788 , 5.77917 , 5.77953 , 5.77988 , 5.78022 , 5.78056 , 5.78088 , 5.7812 , 5.78151 , 5.78182 , 5.78212 , 5.78241 , 5.78269 , 5.78297 , 5.78324 , 5.78351 , 5.78377 , 5.78402 , 5.78427 , 5.78452 , 5.78475 , 5.78499 , 5.78521 , 5.78543 , 5.78565 , 5.78587 , 5.78607 , 5.78628 , 5.78648 , 5.78668 , 5.78687 , 5.78706 , 5.78724 , 5.78743 , 5.78761 , 5.78778 , 5.78796 , 5.78813 , 5.7883 , 5.78846 , 5.78862 , 5.78879 , 5.78895 , 5.7891 , 5.78926 , 5.78941 , 5.78957 , 5.78972 , 5.78987 , 5.79002 , 5.79017 , 5.79032 , 5.79046 , 5.79061 , 5.79076 , 5.79091 , 5.79105 , 5.7912 , 5.79135 , 5.7915 , 5.79164 , 5.79179 , 5.79194 , 5.79209 , 5.79224 , 5.7924 , 5.79255 , 5.79271 , 5.79286 , 5.79302 , 5.79318 , 5.79335 , 5.79351 , 5.79368 , 5.79385 , 5.79402 , 5.79419 , 5.79437 , 5.79455 , 5.79473 , 5.79492 , 5.79511 , 5.7953 , 5.7955 , 5.7957 , 5.7959 , 5.7961 , 5.79631 , 5.79653 , 5.79675 , 5.79697 , 5.7972 , 5.79743 , 5.79766 , 5.7979 , 5.79815 , 5.7984 , 5.79865 , 5.79891 , 5.79918 , 5.79945 , 5.79972 , 5.80001 , 5.80029 , 5.80058 , 5.80088 , 5.80119 , 5.8015 , 5.80181 , 5.80213 , 5.80246 , 5.8028 , 5.80314 , 5.80349 , 5.80384 , 5.8042 , 5.80457 , 5.80494 , 5.80532 , 5.80571 , 5.80611 , 5.80651 , 5.80692 , 5.80734 , 5.80777 , 5.8082 , 5.80864 , 5.80909 , 5.80955 , 5.81001 , 5.81048 , 5.81096 , 5.81145 , 5.81195 , 5.81246 , 5.81297 , 5.81349 , 5.81402 , 5.81456 , 5.81511 , 5.81567 , 5.81623 , 5.81681 , 5.81739 , 5.81799 , 5.81859 , 5.8192 , 5.81982 , 5.82045 , 5.82109 , 5.82174 , 5.8224 , 5.82307 , 5.82374 , 5.82443 , 5.82513 , 5.82584 , 5.82656 , 5.82728 , 5.82802 , 5.82877 , 5.82953 , 5.8303 , 5.83108 , 5.83186 , 5.83266 , 5.83347 , 5.8343 , 5.83513 , 5.83597 , 5.83682 , 5.83769 , 5.83856 , 5.83944 , 5.84034 , 5.84125 , 5.84217 , 5.8431 , 5.84404 , 5.84499 , 5.84595 , 5.84692 , 5.84791 , 5.84891 , 5.84992 , 5.85093 , 5.85197 , 5.85301 , 5.85406 , 5.85513 , 5.85621 , 5.85729 , 5.8584 , 5.85951 , 5.86063 , 5.86177 , 5.86292 , 5.86408 , 5.86525 , 5.86643 , 5.86763 , 5.86884 , 5.87005 , 5.87129 , 5.87253 , 5.87379 , 5.87506 , 5.87634 , 5.87763 , 5.87893 , 5.88025 , 5.88158 , 5.88292 , 5.88428 , 5.88564 , 5.88702 , 5.88841 , 5.88982 , 5.89123 , 5.89266 , 5.8941 , 5.89556 , 5.89702 , 5.8985 , 5.89999 , 5.90149 , 5.90301 , 5.90454 , 5.90608 , 5.90763 , 5.9092 , 5.91078 , 5.91237 , 5.91398 , 5.91559 , 5.91722 , 5.91886 , 5.92052 , 5.92219 , 5.92387 , 5.92556 , 5.92727 , 5.92898 , 5.93071 , 5.93246 , 5.93421 , 5.93598 , 5.93776 , 5.93956 , 5.94137 , 5.94319 , 5.94502 , 5.94686 , 5.94872 , 5.95059 , 5.95247 , 5.95437 , 5.95628 , 5.9582 , 5.96013 , 5.96207 , 5.96403 , 5.966 , 5.96799 , 5.96998 , 5.97199 , 5.97401 , 5.97604 , 5.97809 , 5.98015 , 5.98222 , 5.9843 , 5.98639 , 5.9885 , 5.99062 , 5.99275 , 5.9949 , 5.99706 , 5.99922 , 6.00141 , 6.0036 , 6.0058 , 6.00802 , 6.01025 , 6.01249 , 6.01475 , 6.01701 , 6.01929 , 6.02158 , 6.02388 , 6.02619 , 6.02852 }
bbl = { 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 }
smoothing factors = { 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 }

]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Added tika met key [Content-Type] with value [text/plain; charset=ISO-8859-1]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Added config file met key [ProductType] with value [GenericFile]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.crawl.ProductCrawler ingest
INFO: ProductCrawler: Ready to ingest product: [/usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr]: ProductType: [GenericFile]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.filemgr.ingest.StdIngester setFileManager
INFO: StdIngester: connected to file manager: [http://localhost:9000]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferer setFileManagerUrl
INFO: In Place Data Transfer to: [http://localhost:9000] enabled
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.filemgr.ingest.StdIngester ingest
INFO: StdIngester: ingesting product: ProductName: [ang20150419t155032_corr_v1f_img.hdr]: ProductType: [GenericFile]: FileLocation: [/usr/local/coal-sds-deploy/data/staging/]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.filemgr.versioning.VersioningUtils createBasicDataStoreRefsFlat
FINE: VersioningUtils: Generated data store ref: file:/usr/local/coal-sds-deploy/data/archive/ang20150419t155032_corr_v1f_img.hdr/ang20150419t155032_corr_v1f_img.hdr from origRef: file:/usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.crawl.ProductCrawler ingest
INFO: Successfully ingested product: [/usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr]: product id: c3b4484f-4f7b-4b77-a6db-0bd20b3e117e
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.crawl.ProductCrawler handleFile
INFO: Successful ingest of product: [/usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.crawl.ProductCrawler handleFile
INFO: Handling file /usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr.met
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Invoking tika extractor on file [/usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr.met]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Number of captured tika metadata keys: [4]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Added tika met key [X-Parsed-By] with value [org.apache.tika.parser.DefaultParser]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Added tika met key [Content-Encoding] with value [ISO-8859-1]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Added tika met key [content] with value [<cas:metadata xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas">
</cas:metadata>
]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Added tika met key [Content-Type] with value [text/plain; charset=ISO-8859-1]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor extrMetadata
FINE: Added config file met key [ProductType] with value [GenericFile]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.crawl.ProductCrawler ingest
INFO: ProductCrawler: Ready to ingest product: [/usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr.met]: ProductType: [GenericFile]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.filemgr.ingest.StdIngester ingest
INFO: StdIngester: ingesting product: ProductName: [ang20150419t155032_corr_v1f_img.hdr.met]: ProductType: [GenericFile]: FileLocation: [/usr/local/coal-sds-deploy/data/staging/]
Sep 28, 2018 2:31:32 PM org.apache.oodt.cas.filemgr.versioning.VersioningUtils createBasicDataStoreRefsFlat
FINE: VersioningUtils: Generated data store ref: file:/usr/local/coal-sds-deploy/data/archive/ang20150419t155032_corr_v1f_img.hdr.met/ang20150419t155032_corr_v1f_img.hdr.met from origRef: file:/usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr.met
Sep 28, 2018 2:31:33 PM org.apache.oodt.cas.crawl.ProductCrawler ingest
INFO: Successfully ingested product: [/usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr.met]: product id: 275d2ec2-0c95-453f-8a3f-661cf1a40e98
Sep 28, 2018 2:31:33 PM org.apache.oodt.cas.crawl.ProductCrawler handleFile
INFO: Successful ingest of product: [/usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr.met]

The product is then ingested and looks as follows

      {
        "id": "c3b4484f-4f7b-4b77-a6db-0bd20b3e117e",
        "CAS.ProductStructure": "Flat",
        "CAS.ProductTypeName": "GenericFile",
        "CAS.ProductName": "ang20150419t155032_corr_v1f_img.hdr",
        "CAS.ProductReceivedTime": "2018-09-28T14:31:32Z",
        "CAS.ProductTypeId": "urn:oodt:GenericFile",
        "CAS.ProductTransferStatus": "RECEIVED",
        "CAS.ProductId": "c3b4484f-4f7b-4b77-a6db-0bd20b3e117e",
        "X-Parsed-By": [
          "org.apache.tika.parser.DefaultParser"
        ],
        "ProductName": [
          "ang20150419t155032_corr_v1f_img.hdr"
        ],
        "FileLocation": [
          "/usr/local/coal-sds-deploy/data/archive/ang20150419t155032_corr_v1f_img.hdr"
        ],
        "Content-Encoding": [
          "ISO-8859-1"
        ],
        "Filename": [
          "ang20150419t155032_corr_v1f_img.hdr"
        ],
        "content": [
          "ENVI\ndescription = {\n  Georeferenced Image built from input GLT. [Tue Aug 11 15:39:46 2015] [Tue\n  Aug 11 16:17:51 2015]}\nsamples = 1363\nlines = 22826\nbands = 432\nheader offset = 0\nfile type = ENVI Standard\ndata type = 4\ninterleave = bil\nsensor type = Unknown\nbyte order = 0\nmap info = { UTM , 1.000 , 1.000 , 201603.137 , 4061363.983 , 2.7000000000e+00 , 2.7000000000e+00 , 13 , North , WGS-84 , units=Meters , rotation=90.00000000 }\nwavelength units = Nanometers\ncorrection factors = { 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.880586 , 1.741631 , 1.524632 , 1.564904 , 1.390035 , 1.342501 , 1.264768 , 1.200357 , 1.131471 , 1.06512 , 1.105211 , 1.081234 , 1.064999 , 1.056798 , 1.042573 , 1.009621 , 1.009031 , 1.002107 , 0.997332 , 0.976807 , 0.963755 , 0.969028 , 0.962823 , 0.949522 , 0.960568 , 0.957813 , 0.934766 , 0.944994 , 0.937726 , 0.935257 , 0.932706 , 0.932568 , 0.933217 , 0.928705 , 0.929294 , 0.936669 , 0.935498 , 0.94823 , 0.949846 , 0.945885 , 0.935468 , 0.930084 , 0.934473 , 0.935378 , 0.939193 , 0.935081 , 0.937398 , 0.943396 , 0.947133 , 0.950645 , 0.945531 , 0.940295 , 0.933129 , 0.930664 , 0.92736 , 0.931786 , 0.928217 , 0.928205 , 0.926481 , 0.928583 , 0.930504 , 0.93648 , 0.930731 , 0.931265 , 0.935063 , 0.93434 , 0.926983 , 0.932689 , 0.936477 , 0.939647 , 0.940155 , 0.937519 , 0.939448 , 0.942124 , 0.93653 , 0.9435 , 0.959204 , 0.942566 , 0.940873 , 0.939414 , 0.939822 , 0.940174 , 0.941372 , 0.939347 , 0.942108 , 0.942664 , 0.934811 , 0.934567 , 0.937712 , 0.940611 , 0.944809 , 0.939877 , 0.943376 , 0.939189 , 0.943619 , 0.946268 , 0.940166 , 0.953752 , 0.958975 , 0.954512 , 0.954103 , 0.958978 , 0.953247 , 0.952199 , 0.956082 , 0.957846 , 0.970078 , 0.973704 , 0.980014 , 0.928845 , 0.922973 , 0.954414 , 0.95521 , 0.961276 , 0.964513 , 0.965296 , 0.964644 , 0.954999 , 0.951133 , 0.956216 , 0.951977 , 0.948547 , 0.949499 , 0.952685 , 0.950158 , 0.944263 , 0.936946 , 0.938394 , 0.941325 , 0.94116 , 0.941397 , 0.940811 , 0.942695 , 0.945228 , 0.953929 , 0.962457 , 0.968728 , 0.963947 , 0.961222 , 0.963003 , 0.967658 , 0.969773 , 0.970294 , 0.963456 , 0.970497 , 0.976972 , 0.961611 , 0.953081 , 0.945668 , 0.993867 , 1.019915 , 0.997013 , 0.977643 , 0.998022 , 1.007041 , 1.003881 , 0.991335 , 0.976202 , 0.967636 , 0.969294 , 0.965331 , 0.968705 , 0.965705 , 0.973601 , 0.97282 , 0.970848 , 0.970687 , 0.969394 , 0.972263 , 0.969286 , 0.970327 , 0.97754 , 0.984703 , 0.993916 , 1.02186 , 1.054704 , 1.061183 , 1.036962 , 1.012519 , 0.991209 , 0.975974 , 0.965446 , 0.958801 , 0.951519 , 0.960628 , 0.957276 , 0.96061 , 0.95689 , 0.956666 , 0.965567 , 0.982251 , 1.038526 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.131643 , 1.150333 , 1.112466 , 1.06255 , 1.033182 , 1.03018 , 1.006512 , 0.993285 , 1.003557 , 1.014441 , 0.998586 , 0.994426 , 0.987975 , 0.984681 , 0.981863 , 0.964106 , 0.956432 , 0.954467 , 0.956429 , 0.950668 , 0.947217 , 0.944635 , 0.942942 , 0.941078 , 0.943502 , 0.950408 , 0.957625 , 0.965073 , 0.976012 , 0.976683 , 0.975625 , 0.968011 , 0.968843 , 0.970632 , 0.960977 , 0.960505 , 0.955015 , 0.953597 , 0.951119 , 0.945679 , 0.949988 , 0.951236 , 0.947813 , 0.948004 , 0.950015 , 0.939258 , 0.945863 , 0.953927 , 0.953145 , 0.945291 , 0.942319 , 0.947022 , 0.948264 , 0.947112 , 0.942092 , 0.943128 , 0.948068 , 0.944432 , 0.950396 , 0.964006 , 0.961019 , 0.951786 , 0.957457 , 0.950327 , 0.954375 , 0.9608 , 0.965864 , 0.982396 , 1.011334 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.289798 , 1.126375 , 1.074621 , 1.069365 , 1.051364 , 1.009432 , 0.984578 , 0.991187 , 1.016306 , 1.083681 , 1.076539 , 1.104657 , 1.114005 , 1.081102 , 1.039196 , 1.009524 , 0.966515 , 0.953688 , 0.966274 , 0.973232 , 0.966402 , 0.946801 , 0.951952 , 0.965679 , 0.96977 , 0.946541 , 0.941678 , 0.937528 , 0.922351 , 0.914192 , 0.92879 , 0.932284 , 0.933182 , 0.922322 , 0.91851 , 0.919591 , 0.925027 , 0.924611 , 0.932288 , 0.933352 , 0.930517 , 0.931666 , 0.931763 , 0.932655 , 0.928945 , 0.933308 , 0.932392 , 0.932943 , 0.935328 , 0.947019 , 0.954093 , 0.95156 , 0.939591 , 0.942808 , 0.944862 , 0.944004 , 0.949161 , 0.950992 , 0.956738 , 0.951184 , 0.953545 , 0.958836 , 0.966134 , 0.956752 , 0.951961 , 0.958667 , 0.9579 , 0.968531 , 0.973792 , 0.969238 , 0.970838 , 0.954552 , 0.968166 , 0.989176 , 0.974784 , 0.970674 , 0.9733 , 0.990576 , 1.0062 , 1.010295 , 0.99378 , 0.986109 , 1.007054 , 1.005377 , 1.010013 , 1.014671 , 1.021618 , 1.021229 , 1.021003 , 1.020866 , 1.029358 , 1.042136 , 1.030482 , 1.019556 , 1.036656 , 1.05348 , 1.015947 , 1.07263 , 1.092879 , 1.053624 , 1.086491 , 1.139334 , 1.163645 , 1.162487 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 }\nwavelength = { 346.6325373 , 351.6412073 , 356.6498773 , 361.6585473 , 366.6672173 , 371.6758873 , 376.6845573 , 381.6932273 , 386.7018973 , 391.7105673 , 396.7192373 , 401.7279073 , 406.7365773 , 411.7452473 , 416.7539173 , 421.7625873 , 426.7712573 , 431.7799273 , 436.7885973 , 441.7972673 , 446.8059373 , 451.8146073 , 456.8232773 , 461.8319473 , 466.8406173 , 471.8492873 , 476.8579573 , 481.8666273 , 486.8752973 , 491.8839673 , 496.8926373 , 501.9013073 , 506.9099773 , 511.9186473 , 516.9273173 , 521.9359873 , 526.9446573 , 531.9533273 , 536.9619973 , 541.9706673 , 546.9793373 , 551.9880073 , 556.9966773 , 562.0053473 , 567.0140173 , 572.0226873 , 577.0313573 , 582.0400273 , 587.0486973 , 592.0573673 , 597.0660373 , 602.0747073 , 607.0833773 , 612.0920473 , 617.1007173 , 622.1093873 , 627.1180573 , 632.1267273 , 637.1353973 , 642.1440673 , 647.1527373 , 652.1614073 , 657.1700773 , 662.1787473 , 667.1874173 , 672.1960873 , 677.2047573 , 682.2134273 , 687.2220973 , 692.2307673 , 697.2394373 , 702.2481073 , 707.2567773 , 712.2654473 , 717.2741173 , 722.2827873 , 727.2914573 , 732.3001273 , 737.3087973 , 742.3174673 , 747.3261373 , 752.3348073 , 757.3434773 , 762.3521473 , 767.3608173 , 772.3694873 , 777.3781573 , 782.3868273 , 787.3954973 , 792.4041673 , 797.4128373 , 802.4215073 , 807.4301773 , 812.4388473 , 817.4475173 , 822.4561873 , 827.4648573 , 832.4735273 , 837.4821973 , 842.4908673 , 847.4995373 , 852.5082073 , 857.5168773 , 862.5255473 , 867.5342173 , 872.5428873 , 877.5515573 , 882.5602273 , 887.5688973 , 892.5775673 , 897.5862373 , 902.5949073 , 907.6035773 , 912.6122473 , 917.6209173 , 922.6295873 , 927.6382573 , 932.6469273 , 937.6555973 , 942.6642673 , 947.6729373 , 952.6816073 , 957.6902773 , 962.6989473 , 967.7076173 , 972.7162873 , 977.7249573 , 982.7336273 , 987.7422973 , 992.7509673 , 997.7596373 , 1002.7683073 , 1007.7769773 , 1012.7856473 , 1017.7943173 , 1022.8029873 , 1027.8116573 , 1032.8203273 , 1037.8289973 , 1042.8376673 , 1047.8463373 , 1052.8550073 , 1057.8636773 , 1062.8723473 , 1067.8810173 , 1072.8896873 , 1077.8983573 , 1082.9070373 , 1087.9157073 , 1092.9243673 , 1097.9330373 , 1102.9417073 , 1107.9503873 , 1112.9590473 , 1117.9677273 , 1122.9763873 , 1127.9850573 , 1132.9937273 , 1138.0024073 , 1143.0110673 , 1148.0197373 , 1153.0284073 , 1158.0370873 , 1163.0457473 , 1168.0544273 , 1173.0630973 , 1178.0717673 , 1183.0804373 , 1188.0891073 , 1193.0977773 , 1198.1064473 , 1203.1151173 , 1208.1237873 , 1213.1324573 , 1218.1411273 , 1223.1497973 , 1228.1584673 , 1233.1671373 , 1238.1758073 , 1243.1844773 , 1248.1931473 , 1253.2018173 , 1258.2104873 , 1263.2191573 , 1268.2278273 , 1273.2364973 , 1278.2451673 , 1283.2538373 , 1288.2625073 , 1293.2711773 , 1298.2798473 , 1303.2885173 , 1308.2971873 , 1313.3058573 , 1318.3145273 , 1323.3231973 , 1328.3318673 , 1333.3405373 , 1338.3492073 , 1343.3578773 , 1348.3665473 , 1353.3752173 , 1358.3838873 , 1363.3925573 , 1368.4012273 , 1373.4098973 , 1378.4185673 , 1383.4272373 , 1388.4359073 , 1393.4445773 , 1398.4532473 , 1403.4619173 , 1408.4705873 , 1413.4792573 , 1418.4879273 , 1423.4965973 , 1428.5052673 , 1433.5139373 , 1438.5226073 , 1443.5312773 , 1448.5399473 , 1453.5486173 , 1458.5572873 , 1463.5659573 , 1468.5746273 , 1473.5832973 , 1478.5919673 , 1483.6006373 , 1488.6093073 , 1493.6179773 , 1498.6266473 , 1503.6353173 , 1508.6439873 , 1513.6526573 , 1518.6613273 , 1523.6699973 , 1528.6786673 , 1533.6873373 , 1538.6960073 , 1543.7046773 , 1548.7133473 , 1553.7220173 , 1558.7306873 , 1563.7393573 , 1568.7480273 , 1573.7566973 , 1578.7653673 , 1583.7740373 , 1588.7827073 , 1593.7913773 , 1598.8000473 , 1603.8087173 , 1608.8173873 , 1613.8260573 , 1618.8347273 , 1623.8433973 , 1628.8520673 , 1633.8607373 , 1638.8694073 , 1643.8780773 , 1648.8867473 , 1653.8954173 , 1658.9040873 , 1663.9127573 , 1668.9214273 , 1673.9300973 , 1678.9387673 , 1683.9474373 , 1688.9561073 , 1693.9647773 , 1698.9734473 , 1703.9821173 , 1708.9907873 , 1713.9994573 , 1719.0081273 , 1724.0167973 , 1729.0254673 , 1734.0341373 , 1739.0428073 , 1744.0514773 , 1749.0601473 , 1754.0688173 , 1759.0774873 , 1764.0861573 , 1769.0948273 , 1774.1034973 , 1779.1121673 , 1784.1208373 , 1789.1295073 , 1794.1381773 , 1799.1468473 , 1804.1555173 , 1809.1641873 , 1814.1728573 , 1819.1815273 , 1824.1901973 , 1829.1988673 , 1834.2075373 , 1839.2162073 , 1844.2248773 , 1849.2335473 , 1854.2422173 , 1859.2508873 , 1864.2595573 , 1869.2682273 , 1874.2768973 , 1879.2855673 , 1884.2942373 , 1889.3029073 , 1894.3115773 , 1899.3202473 , 1904.3289173 , 1909.3375873 , 1914.3462573 , 1919.3549373 , 1924.3635973 , 1929.3722773 , 1934.3809373 , 1939.3896073 , 1944.3982873 , 1949.4069573 , 1954.4156173 , 1959.4242873 , 1964.4329673 , 1969.4416373 , 1974.4502973 , 1979.4589673 , 1984.4676373 , 1989.4763173 , 1994.4849773 , 1999.4936473 , 2004.5023273 , 2009.5109973 , 2014.5196673 , 2019.5283373 , 2024.5370073 , 2029.5456773 , 2034.5543473 , 2039.5630173 , 2044.5716873 , 2049.5803573 , 2054.5890273 , 2059.5976973 , 2064.6063673 , 2069.6150373 , 2074.6237073 , 2079.6323773 , 2084.6410473 , 2089.6497173 , 2094.6583873 , 2099.6670573 , 2104.6757273 , 2109.6843973 , 2114.6930673 , 2119.7017373 , 2124.7104073 , 2129.7190773 , 2134.7277473 , 2139.7364173 , 2144.7450873 , 2149.7537573 , 2154.7624273 , 2159.7710973 , 2164.7797673 , 2169.7884373 , 2174.7971073 , 2179.8057773 , 2184.8144473 , 2189.8231173 , 2194.8317873 , 2199.8404573 , 2204.8491273 , 2209.8577973 , 2214.8664673 , 2219.8751373 , 2224.8838073 , 2229.8924773 , 2234.9011473 , 2239.9098173 , 2244.9184873 , 2249.9271573 , 2254.9358273 , 2259.9444973 , 2264.9531673 , 2269.9618373 , 2274.9705073 , 2279.9791773 , 2284.9878473 , 2289.9965173 , 2295.0051873 , 2300.0138573 , 2305.0225273 , 2310.0311973 , 2315.0398673 , 2320.0485373 , 2325.0572073 , 2330.0658773 , 2335.0745473 , 2340.0832173 , 2345.0918873 , 2350.1005573 , 2355.1092273 , 2360.1178973 , 2365.1265673 , 2370.1352373 , 2375.1439073 , 2380.1525773 , 2385.1612473 , 2390.1699173 , 2395.1785873 , 2400.1872573 , 2405.1959273 , 2410.2045973 , 2415.2132673 , 2420.2219373 , 2425.2306073 , 2430.2392773 , 2435.2479473 , 2440.2566173 , 2445.2652873 , 2450.2739573 , 2455.2826273 , 2460.2912973 , 2465.2999673 , 2470.3086373 , 2475.3173073 , 2480.3259773 , 2485.3346473 , 2490.3433173 , 2495.3519873 , 2500.3606573 , 2505.3693273 }\nfwhm = { 5.55165 , 5.55524 , 5.55879 , 5.5623 , 5.56577 , 5.5692 , 5.5726 , 5.57595 , 5.57927 , 5.58255 , 5.58579 , 5.58899 , 5.59216 , 5.59529 , 5.59839 , 5.60144 , 5.60446 , 5.60745 , 5.6104 , 5.61332 , 5.61619 , 5.61904 , 5.62185 , 5.62463 , 5.62737 , 5.63008 , 5.63275 , 5.6354 , 5.63801 , 5.64058 , 5.64313 , 5.64564 , 5.64812 , 5.65057 , 5.65299 , 5.65537 , 5.65773 , 5.66005 , 5.66235 , 5.66461 , 5.66684 , 5.66905 , 5.67122 , 5.67337 , 5.67549 , 5.67758 , 5.67964 , 5.68167 , 5.68367 , 5.68565 , 5.6876 , 5.68952 , 5.69141 , 5.69328 , 5.69512 , 5.69694 , 5.69873 , 5.70049 , 5.70223 , 5.70394 , 5.70563 , 5.70729 , 5.70893 , 5.71055 , 5.71214 , 5.7137 , 5.71524 , 5.71676 , 5.71826 , 5.71973 , 5.72118 , 5.72261 , 5.72401 , 5.72539 , 5.72676 , 5.72809 , 5.72941 , 5.73071 , 5.73198 , 5.73324 , 5.73447 , 5.73569 , 5.73688 , 5.73806 , 5.73921 , 5.74034 , 5.74146 , 5.74256 , 5.74363 , 5.74469 , 5.74573 , 5.74676 , 5.74776 , 5.74875 , 5.74972 , 5.75067 , 5.7516 , 5.75252 , 5.75342 , 5.75431 , 5.75518 , 5.75603 , 5.75687 , 5.75769 , 5.75849 , 5.75928 , 5.76006 , 5.76082 , 5.76156 , 5.76229 , 5.76301 , 5.76371 , 5.7644 , 5.76507 , 5.76573 , 5.76638 , 5.76702 , 5.76764 , 5.76825 , 5.76884 , 5.76943 , 5.77 , 5.77056 , 5.77111 , 5.77164 , 5.77217 , 5.77268 , 5.77319 , 5.77368 , 5.77416 , 5.77463 , 5.77509 , 5.77554 , 5.77598 , 5.77641 , 5.77683 , 5.77725 , 5.77765 , 5.77804 , 5.77843 , 5.7788 , 5.77917 , 5.77953 , 5.77988 , 5.78022 , 5.78056 , 5.78088 , 5.7812 , 5.78151 , 5.78182 , 5.78212 , 5.78241 , 5.78269 , 5.78297 , 5.78324 , 5.78351 , 5.78377 , 5.78402 , 5.78427 , 5.78452 , 5.78475 , 5.78499 , 5.78521 , 5.78543 , 5.78565 , 5.78587 , 5.78607 , 5.78628 , 5.78648 , 5.78668 , 5.78687 , 5.78706 , 5.78724 , 5.78743 , 5.78761 , 5.78778 , 5.78796 , 5.78813 , 5.7883 , 5.78846 , 5.78862 , 5.78879 , 5.78895 , 5.7891 , 5.78926 , 5.78941 , 5.78957 , 5.78972 , 5.78987 , 5.79002 , 5.79017 , 5.79032 , 5.79046 , 5.79061 , 5.79076 , 5.79091 , 5.79105 , 5.7912 , 5.79135 , 5.7915 , 5.79164 , 5.79179 , 5.79194 , 5.79209 , 5.79224 , 5.7924 , 5.79255 , 5.79271 , 5.79286 , 5.79302 , 5.79318 , 5.79335 , 5.79351 , 5.79368 , 5.79385 , 5.79402 , 5.79419 , 5.79437 , 5.79455 , 5.79473 , 5.79492 , 5.79511 , 5.7953 , 5.7955 , 5.7957 , 5.7959 , 5.7961 , 5.79631 , 5.79653 , 5.79675 , 5.79697 , 5.7972 , 5.79743 , 5.79766 , 5.7979 , 5.79815 , 5.7984 , 5.79865 , 5.79891 , 5.79918 , 5.79945 , 5.79972 , 5.80001 , 5.80029 , 5.80058 , 5.80088 , 5.80119 , 5.8015 , 5.80181 , 5.80213 , 5.80246 , 5.8028 , 5.80314 , 5.80349 , 5.80384 , 5.8042 , 5.80457 , 5.80494 , 5.80532 , 5.80571 , 5.80611 , 5.80651 , 5.80692 , 5.80734 , 5.80777 , 5.8082 , 5.80864 , 5.80909 , 5.80955 , 5.81001 , 5.81048 , 5.81096 , 5.81145 , 5.81195 , 5.81246 , 5.81297 , 5.81349 , 5.81402 , 5.81456 , 5.81511 , 5.81567 , 5.81623 , 5.81681 , 5.81739 , 5.81799 , 5.81859 , 5.8192 , 5.81982 , 5.82045 , 5.82109 , 5.82174 , 5.8224 , 5.82307 , 5.82374 , 5.82443 , 5.82513 , 5.82584 , 5.82656 , 5.82728 , 5.82802 , 5.82877 , 5.82953 , 5.8303 , 5.83108 , 5.83186 , 5.83266 , 5.83347 , 5.8343 , 5.83513 , 5.83597 , 5.83682 , 5.83769 , 5.83856 , 5.83944 , 5.84034 , 5.84125 , 5.84217 , 5.8431 , 5.84404 , 5.84499 , 5.84595 , 5.84692 , 5.84791 , 5.84891 , 5.84992 , 5.85093 , 5.85197 , 5.85301 , 5.85406 , 5.85513 , 5.85621 , 5.85729 , 5.8584 , 5.85951 , 5.86063 , 5.86177 , 5.86292 , 5.86408 , 5.86525 , 5.86643 , 5.86763 , 5.86884 , 5.87005 , 5.87129 , 5.87253 , 5.87379 , 5.87506 , 5.87634 , 5.87763 , 5.87893 , 5.88025 , 5.88158 , 5.88292 , 5.88428 , 5.88564 , 5.88702 , 5.88841 , 5.88982 , 5.89123 , 5.89266 , 5.8941 , 5.89556 , 5.89702 , 5.8985 , 5.89999 , 5.90149 , 5.90301 , 5.90454 , 5.90608 , 5.90763 , 5.9092 , 5.91078 , 5.91237 , 5.91398 , 5.91559 , 5.91722 , 5.91886 , 5.92052 , 5.92219 , 5.92387 , 5.92556 , 5.92727 , 5.92898 , 5.93071 , 5.93246 , 5.93421 , 5.93598 , 5.93776 , 5.93956 , 5.94137 , 5.94319 , 5.94502 , 5.94686 , 5.94872 , 5.95059 , 5.95247 , 5.95437 , 5.95628 , 5.9582 , 5.96013 , 5.96207 , 5.96403 , 5.966 , 5.96799 , 5.96998 , 5.97199 , 5.97401 , 5.97604 , 5.97809 , 5.98015 , 5.98222 , 5.9843 , 5.98639 , 5.9885 , 5.99062 , 5.99275 , 5.9949 , 5.99706 , 5.99922 , 6.00141 , 6.0036 , 6.0058 , 6.00802 , 6.01025 , 6.01249 , 6.01475 , 6.01701 , 6.01929 , 6.02158 , 6.02388 , 6.02619 , 6.02852 }\nbbl = { 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 0.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 }\nsmoothing factors = { 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 , 1.0 }\n\n"
        ],
        "MimeType": [
          "application/octet-stream",
          "application",
          "octet-stream"
        ],
        "FileSize": [
          "20732"
        ],
        "Content-Type": [
          "text/plain; charset=ISO-8859-1"
        ],
        "CAS.ReferenceMimeType": [
          "application/octet-stream"
        ],
        "CAS.ReferenceDatastore": [
          "file:/usr/local/coal-sds-deploy/data/archive/ang20150419t155032_corr_v1f_img.hdr/ang20150419t155032_corr_v1f_img.hdr"
        ],
        "CAS.ReferenceFileSize": [
          20732
        ],
        "CAS.ReferenceOriginal": [
          "file:/usr/local/coal-sds-deploy/data/staging/ang20150419t155032_corr_v1f_img.hdr"
        ],
        "_version_": 1612888452999676000
      },

As you can see this is much better but it is not quite what we want for example

  • We want a link to the canonical resource e.g. the remote resource from where the file was originally obtained,
  • we do not want all of the .hdf metadata crammed into one field called content, we really want each field within the .hdr file extracted out into a top level field within the resulting file manager record.
  • additionally, the file is being detected as type octet stream, which is incorrect. It should be detected as type envi.
    If we can address the above issues we will be doing well.

@lewismc
Copy link
Member Author

lewismc commented Sep 28, 2018

We need to update to a newer version of Apache Tika in order to achieve the above. Right now we use Apache Tika versions

./crawler/lib/tika-core-1.10.jar
./crawler/lib/tika-parsers-1.7.jar

We should be using the most recent version 1.19

@lewismc lewismc mentioned this issue Oct 4, 2018
Closed
@DenimMazuki DenimMazuki mentioned this issue Oct 15, 2018
Closed
@lewismc
Copy link
Member Author

lewismc commented Oct 19, 2018

This can be achieved as follows

./crawler_launcher 
  --filemgrUrl http://localhost:9000 
  --operation 
  --launchMetCrawler 
  --clientTransferer org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory 
  --productPath /usr/local/coal-sds-deploy/data/staging 
  --metExtractor org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor 
  --metExtractorConfig /usr/local/coal-sds-deploy/data/met/tika.conf 
  --failureDir /usr/local/coal-sds-deploy/data/failure/ 
  --daemonPort 9003 
  --daemonWait 2 
  --successDir /usr/local/coal-sds-deploy/data/archive/ 
  --actionIds DeleteDataFile

I would like someone to write this up so we will discuss this tomorrow.

@lewismc
Copy link
Member Author

lewismc commented Oct 19, 2018

This now exists at https://github.com/capstone-coal/coal-sds/wiki/Automation-of-Data-Acquisition-and-Crawling

Thank you folks :)

@lewismc lewismc closed this as completed Oct 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lewismc lewismc

Labels
crawler OODT Crawler Component enhancement filemgr OODT File Manager Service help wanted
Projects
None yet
Milestone
0.1
Development

No branches or pull requests
2 participants
@lewismc @thomkenn