Skip to content

Fetching Documents

Jump to bottom
Marios Papachristou edited this page Aug 18, 2019 · 11 revisions

Fetching Documents

Fetching documents via the fetcher.py command line tool

There is the fetcher.py tool with which you can fetch documents providing the start date, end date and output directory. It uses the selenium package and the Chromium Driver. The tool is located under scripts/fetcher.py.

The script requires the Chrome Driver in order to simulate a browser environment to fetch the needed documents from the ET. You can provide the chromedriver executable using the --chromedriver argument at the command line. For an easy-to-install solution visit the Installation Page.

Usage:

$ fetcher.py -h
usage: fetcher.py [-h] -date_from DATE_FROM -date_to DATE_TO -output_dir
                  OUTPUT_DIR [--chromedriver CHROMEDRIVER] [--upload]
                  [--type TYPE]

This is the fetching tool for downloading Government Gazette Issues from the
ET. For more information visit
https://github.com/eellak/gsoc2018-3gm/wiki/Fetching-Documents

optional arguments:
  -h, --help            show this help message and exit

required arguments:
  -date_from DATE_FROM  Date from in DD.MM.YYYY format
  -date_to DATE_TO      Date to in DD.MM.YYYY format
  -output_dir OUTPUT_DIR
                        Output Directory

optional arguments:
  --chromedriver CHROMEDRIVER
                        Chrome driver executable
  --upload              Upload to database
  --type TYPE           Government Gazette document type (Teychos)

You need to provide the tool with the start and end date in DD.MM.YYYY format and the output directory of the documents. For example:

python3 fetcher.py -date_from 17.06.2018 -date_to 19.06.2018 -output_dir ./issues --chromedriver /usr/lib/chromium-browser/chromedriver

Scheduling Document Fetching

For a continuous integration scheme, it is advised to fetch documents from the ET on a daily basis. This can be done using the scripts/fetch_daily.sh script that downloads documents on a daily basis. It can be used along with the cron tool in order for the task to be scheduled. A workaround for scheduling is shown below:

  1. To edit your cron configuration:
crontab -e

Add this command line:

30 2 * * * /path-to-project/scripts/fetch_daily.sh /output/dir

The crontab parses the following pattern:

MIN HOUR DOM MON DOW CMD

The values allowed are the following:

Format Meanings and Allowed Value:
MIN     Minute field    0 to 59
HOUR    Hour field      0 to 23
DOM     Day of Month    1-31
MON     Month field     1-12
DOW     Day Of Week     0-6
CMD     Command     Any command to be executed.

The daemon may need restart. For Debian-based systems:

sudo service crond restart

Internet Archive collection

To make Greek Government Gazette issues more accessible we have created a collection on the Internet Archive that currently contains around 130,000 issues of various issue types, spaning from 1976 to 2019. You can access it, perform queries and download issues using the Internet Archive API for python.

A Simple Guide to mining the Greek Government Gazette

A great part of the GSOC-2019 project involved working with great amounts of data from the ET website. In this segment we would like to document the process of data-mining the website and then explain how we can use the scripts included in the project code to efficiently download large corpora of GGG issues. Note that this project concerned GGG issues published strictly after the “Metapoliteusi”(the reinstitution of the current Hellenic republic) and therefore concerns publications after 1976 exclusively.

The only way to batch download issues of the GGG is through he ET website using the page “Anazitiseis FEK”. In order to use the module we have to specify the “year” of the issue’s publication and subsequently the type of issue we would like to download. There are a number of current and discontinued issues we can download and these include:

Issue name Currently used Contents
ΠΡΩΤΟ (Α) Yes Laws, amendments, presidential decrees
ΔΕΥΤΕΡΟ (Β) Yes Mainly administrative decisions
ΤΡΙΤΟ (Γ) Yes Mainly public position offers and appointments
ΤΕΤΑΡΤΟ (Δ) Yes Mainly acts concerning public property
Ανωτάτου Ειδικού Δικαστηρίου (Α.ΕΙ.Δ) Yes Decrees of the Supreme Special Court
Προκηρύξεων Ανωτάτου Συμβουλίου Επιλογής Προσωπικού (Α.Σ.Ε.Π.) Yes Decrees on personnel hirings in the public sector
(ΠΡΑ.Δ.Ι.Τ.) Yes Figures of private and public orgs
Διακηρύξεων Δημοσίων Συμβάσεων (Δ.Δ.Σ.) Yes Summaries of “Declarations of Public Contracts”
(Υ.Ο.Δ.Δ.) Yes Mainly decrees concerning directors and administrative personnel of public organisations
Αναγκαστικών Απαλλοτριώσεων και Πολεοδομικών Θεμάτων (Α.Α.Π.) Yes Expropriations and urban planning
Νομικών Προσώπων Δημοσίου Δικαίου (Ν.Π.Δ.Δ.) No Personal decrees and appointments concerning

Ν.Π.Δ.Δ.s

Αναπτυξιακών Πράξεων και Συμβάσεων (Α.Π.Σ.) No Economic development decrees and contracts
Παράρτημα No Various tables
Εμπορικής και Βιομηχανικής Ιδιοκτησίας (Ε.Β.Ι.) No

Unfortunately there are no statistics or figures concerning the number of issues published per type and year and information concerning issuesthemselves is somewhat restricted. In this guide we provide some simple statistics that can save time for a dataminer.

Issue abbreviation Year of start Year of end
Α’ 1976 2019
Β’ 1976 2019
Γ’ 1983 2019
Δ’ 1976 2019
Α.ΕΙ.Δ 2000 2019
Α.Σ.Ε.Π. 2000 2019
ΠΡΑ.Δ.Ι.Τ. 2016 2019
Δ.Δ.Σ. 2000 2019
Υ.Ο.Δ.Δ. 2006 2019
Α.Α.Π. 2016 2019
Παράρτημα

The ET website has put a cap on how many requests it can service and that means we have to regulate our queries in order to avoid HTTP:503 errors. A way to do this is by using cron or adding sleep commands between queries in bash. Furthermore ET can only present up to 200 issues per query which makes batch downloading somewhat tricky. To make things easier I created a variation of the fetching script form 2018 that permits searching by issue number as opposed to issue date. This script can be fairly easily used in bash to mine the website.

For example let’s say we want to mine all type D issues from year 1984. We can navigate to the respective page on ET and query for this issue. We see that there are slightly more than 800 results. We can then write a very simple bash script to mine all these issues. This example can be found in the scripts directory.

#!/bin/bash

for (( i=1; i<=1000; i=i+200 ))

do

end=$(($i+199))

python3 fetch_by_issue.py -issue_from $i -issue_to $end -year 1984 -output_dir ./issues \

--chromedriver chromedriver --type  Δ

sleep 5

done
Clone this wiki locally