TODO

General
-------


XML parser
----------

- Convert namespaces to existing atoms?

- Add BOM handling

- Continuation must be handled character by character. For example need
  to be careful with <<"\r">>, <<"\n">>

- Check character references for valid range?

- Add callback for ignorable whitespace?

- Parse entity declarations?

- Parse/ignore <!DOCTYPE

- Allow start_element callback say to parse tag content as characters? Need
  to be careful with CDATA sub-elements.

- Add independent callback interface (or just function) to request decoder?


FeedParser
----------

- Store unknown tags as list?


HTTP client
-----------

- Handle Transfer-Encoding headers (see also HTTP/1.1 support below). Also
  ignore Content-Length if Transfer-Encoding is present.

- Add support for methods other than GET and HEAD. Maybe behaviour interface
  also need to be updated for example to transfer big files with POST method.

- Add support for different content codings

- HTTP/1.1 support: chunked transfer coding, persistent and pipeline
  connections.

- HTTP version switch

- HTTPS support

- TCP6 support?


Crawler
-------

- Store URLs in database (we can store hosts in inverse form, like
  com.google.www

- Add timeout for every crawl step and restart timed out process if needed

- Add limit on concurrent processes

- Pause creation of new subprocess after abnormal subprocess termination???