How to develop

How to develop

Parser

How to run parser

sudo docker pull foodpedia/goodsmatrix_parser
touch "$(pwd)"/result.ttl
sudo docker run --rm \
-e "yandexapikey=<yandex_translate_api_key>" \
-v "$(pwd)"/result.ttl:/upload/dump.ttl \
foodpedia/goodsmatrix_parser <category>

# if the category is big then it make sense to up local agrovoc endpoint:
sudo docker build -t local-agrovoc ./foodpedia-parser-v2/AGROVOC-products-substances/
sudo docker run -d --name local-agrovoc local-agrovoc

sudo docker pull foodpedia/goodsmatrix_parser
touch "$(pwd)"/result.ttl
sudo docker run --rm \
-e "yandexapikey=<yandex_translate_api_key>" \
-e "agrovoc_endpoint=http://local-agrovoc:3030/agrovoc/query" \
--link local-agrovoc:local-agrovoc \
-v "$(pwd)"/result.ttl:/upload/dump.ttl \
foodpedia/goodsmatrix_parser <category>

Use category's name from goodsmatrix's URL.

The yandex_translate_api_key is used to translate food's name and description to English with help of Yandex Translate API. You can find it in Basecamp: https://basecamp.com/2517633/projects/5593591/documents/6333756

example: how to parse the Milk category

Milk products category's URL: http://goodsmatrix.ru/goods-catalogue/Milk.html

touch milk.ttl
sudo docker run --rm \
-e "yandexapikey=<yandex_translate_api_key>" \
-v "$(pwd)"/milk.ttl:/upload/dump.ttl \
foodpedia/goodsmatrix_parser Milk

Development cycle for parser

#1. Take the latest sources and docker container
git clone [email protected]:ailabitmo/foodpedia.git && cd ./foodpedia
sudo docker pull foodpedia/goodsmatrix_parser
#2. Make your changes
#3. Build image
sudo docker build -t foodpedia/goodsmatrix_parser ./foodpedia-parser-v2
#4. Run unit tests:
sudo docker run --rm \
-v "$(pwd)"/foodpedia-parser-v2/:/usr/local/src/foodpedia-parser-v2/ \
foodpedia/goodsmatrix_parser -u
#5. Run manual integration test
touch integration_test_result.ttl
sudo docker run --rm \
-e "yandexapikey=<yandex_translate_api_key>" \
-v "$(pwd)"/foodpedia-parser-v2/:/usr/local/src/foodpedia-parser-v2/ \
-v "$(pwd)"/integration_test_result.ttl:/upload/dump.ttl \
foodpedia/goodsmatrix_parser <category>
#Check the integration_test_result.ttl`
#6. If tests passed, commit changes
#7. Check that the Jenkins job [foodpedia_parser_build_docker_image](http://109.234.34.200:8080/job/foodpedia_parser_build_docker_image/) finished successfully
#8. Take the latest image from docker hub
sudo docker pull foodpedia/goodsmatrix_parser