Pipermail getting script

[CorneJB]

Hiya!

I found the following script (https://mail.python.org/pipermail/mailman-users/2012-October/074208.html) that with some modifications behaves very similar to mbox_downloader.sh but for pipermail projects. The resulting mbox file seems to be usable in kaiaulu in the same manner as the ones downloaded by mbox_downloader.sh.

 #!/bin/sh
  # automated retrieval of pipermail archives & conversion to mbox file
  # Last edit: 2021 29/4/2021 17:42
  listname=$(echo "$1" | sed 's:^\(http.*\)/\([^/]*\)/$:\2:')
  current_dir=$PWD
  cd /tmp
  wget -r -l 1 -nH -A *.txt.gz "$1"
  touch /tmp/pipermail/$listname/$listname.mbox
  chmod 600 /tmp/pipermail/$listname/$listname.mbox
  cd /tmp/pipermail/$listname
  for f in $(ls |sort)
  do zcat $f | iconv -f iso8859-15 -t utf-8 | sed 's/\(^From.*\)\ at\ /\1@/' >> "$listname.mbox"
  done
  cd $current_dir
  mv /tmp/pipermail/$listname/$listname.mbox ./
  rm /tmp/pipermail/$listname/*.gz

Kind regards,

Corne Broere

[carlosparadis]

Nice finding @CorneJB! In the same e-mail post it seems there is also another one in https://github.com/wesleyd/pipermail-archive-to-maildir being cited. Were you able to successfully run it against any pipermail project mailing list?

I and @harismumtaz18 were just discussing about mailing lists yesterday, so I think this will be of his interest. This expands what mailing lists we can reach! For Google Groups I was able to use https://gitlab.com/mcepl/gg_scraper on the OpenSSL project before.

[carlosparadis]

For projects that only use issue trackers for discussion, which I think is unlikely we find anything to convert to .mbox, these should be doable to use too:

• JIRA: https://github.com/matbmeijer/JirAgileR/
• GitHub: https://github.com/r-lib/gh

The equivalent object returned for comments should be of the same granularity (i.e. instead of every row being an e-mail reply, it will be an issue comment). So using the social smells notebook, and loading a table with the same column names instead of using project_mbox variable should do the trick.

[CorneJB]

Also being able to do issue trackers sounds amazing. Over the weekend I made the pyperget python script from the piperget shell script. It is agnostic to the path and the compression of the pipermail archives. When loading the .mbox it produces into Kaiaulu, the smells notebook seems to work fine. However in comparing it to a native .mbox format from an Apache project, I end up with just 19 columns (compared to the 429 on the native .mbox). Maybe I just got lucky and the 19 columns contain the data that is needed for analysis, but I'm not really sure.

Here is the link to the tool I adapted from the original script: https://github.com/CorneJB/pyperget/blob/master/pyperget.py

[carlosparadis]

Would you have any interest in trying to convert the above into an R function to pull request into the codebase? I think the equivalent to wget in R would be httr https://httr.r-lib.org. I can open an issue for us to iterate if you like.

[CorneJB]

My experience with R is not that vast but I would definitely like to try!

[carlosparadis]

Great! Let me open an issue then so we can iterate there :) You can ask me anything there if you get stuck, I'd be more than happy to help.