Experimental rsget.pl core. A shape of new things to come.

WARNING: my vision of the core has changed somewhat since writing this
document.


The goal is to adopt rsget.pl to server-like environment, possibly destroying
any compatibility with earlier versions.


Remarks:
- only daemon mode
- xmlrpc or similar communication method
  -- I'm considering rewriting RPC::PlServer not to use Net::Daemon, but
    IO::Socket directly (either UNIX or INET). RPC::PlClient may need updating.
- no integrated input/output interfaces
- store data in barkeley db, or gnu db
  -- SQL was not a good idea
- multi-user support


Salvaged clients (not part of core):
- output-only terminal visualization
- input/output list file writer (with syntax similar to actual file list,
  or simpler one)
- input/output perl cgi application to create webbrowser interface

Clients planned in future (not part of core):
- I/O ncurses client (something like rtorrent, caca for captcha ?)
	see scraps/showimage.pl
- I/O readline client, with captcha displaying !
- I/O gtk client (try to use code from slimrat)
- port cgi to php (for simplicity)
  -- probably very complicated (thugh not impossible) with storable-based RPC
    communication
- port cgi to mod_perl (for speed)
- kde/gnome tray icon (inform about completed downloads, ask captcha)
  -- current gersion alreadt has tray icon for captcha written in gtk,
    most of it could be reused


Other ideas:
- create http plugin repository, to easily update plugins without
  svn client
  -- http://rsget.pl/download/plugins
- store downloaded plugins in database
- simple text-based communication (could be useful for some more advanced
    hooks)
- (direct download) If URI is not of any supported service, send http head
  request. In case of mime type other than text/* or image/* allow
  downloading the file.
  -- current version already supports it
- (direct ftp download) Any ftp:// URI can be accessed directly only
  -- current version already supports it
- filename_fix -- function similar to unify, but fixes file name returned
  by Content-Disposition (e.g. remove .html_ from RS names)
- rewrite Plugin Processor to use Filter::Util::Call
- full XDG support


* $ rsget.pl --help
usage: rsget.pl [OPTS] COMMAND1 [OPTS1] [ARGS1] -- COMMAND2 [OPTS2] [ARGS2]

Common options:
   --local=FILE           Use UNIX socket for communication
   --remote=[HOST:]PORT   Use TCP for communication

Commands are:
   addlink    Add links from command line to download links
   addlist    Add links from list file to download links
   cgi        CGI interface
   core       Start daemon
   curses     Curses interface
   gtk        GTK interface
   readline   Readline interface
   tray       Tray icon
See 'rsget.pl --help [command]' or 'rsget.pl --help-all' for more information.

'addlink' help:
   Arguments: <link1> <link2> <link3a> + <link3b> + <link3c> <link4>

'addlist' help:
   Arguments: <file1> <flie2>

* rsget.pl command should be a wrapper calling core and/or clients and
  interconnecting them internally or externally

- start daemon:
  $ rsget.pl core --fork

- add link to daemon:
  $ rsget.pl addlink <http://...>
   or
  $ rsget.pl add <http://...>

- add link list to daemon:
  $ rsget.pl addlist <file.list>
   or
  $ rsget.pl add <file.list>

- start core (do not fork), and download some links, (exit after):
  $ rsget.pl core -- add <http://..>

- start curses client and connect to daemon:
  $ rsget.pl curses

- start curses with core:
  $ rsget.pl core -- curses

- most clients and core should be able to coexist
  $ rsget.pl --local core -- addlink <link1> <link2> -- addlist <file.list> -- gtk


* some TODO found lying around:

# Curl.pm: try making servers think we're just a proxy for some other host
#  -- unlikely this is going to work.
#   X-Forwarded-For: XX.XX.XXX.XXX
#   Cache-Control: bypass-client=XX.XX.XX.XXX



* configuration:
- first cmdline arguments are read and based on it config file is opened
- config file is opened and read but cmdline arguments aren't overwritten
- in case of core: read options from database and replace config file arguments
- after that values can be changed manually, those are saved in database

 priority:
  1. changed values
  2. command line arguments
  3. database options
  4. config file
  5. defaults


* communication methods:
  There are multiple methods for communication between core and clients.
- direct -- both core and client are started as one process. Client code is
	executed by core every certain amount of time. Client must not block.
	e.g. clients: addlink, addlist, non-interactive console output

- internal -- core and client are started at the same time, but as different
	threads. They communicate with each other using shared memory.
	e.g. clients: tk, readline, curses
		-- those use event loop, so direct comm isn't possible

- local -- client communicates with already running core using unix socket.
	e.g. clients: cgi
		-- cgi runs in bursts, so core cannot be attached to it

- remote -- client communicates over tcp with core running somewhere else.
	e.g. clients: any of the above, allowing remote cgi communication
		could be dangerous though

 If multiple communication methods are configured in core it tries to allow
communication over any of them.
 Client tries comm starting with best method specified and descends until
finding one that works. Order is: direct (best), internal, local, remote.


* starting new download
 must take into account:
  - downloads running right now
  - available interfaces
  - used getter (or service identifier or available getter)
  - user quota (for that service)
  - whether there are other uris for this file
  - whether service supports continuing downloads

 special considerations:
  - some user may have his own premium account, not shared with other users
  - interfaces may be different for each users (should I allow this?)
  - getter may support continuing download on some files only

 and on top of all that, it must be quick !