Sitemap generator
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
Kamo Petrosyan 177bdb618c readme rst 4 years ago
pysitemap docstring and comments for code 4 years ago
tests mechanize removed 6 years ago
.gitignore docstring and comments for code 4 years ago
CANGELOG новый файл: CANGELOG 6 years ago
LICENSE Update LICENSE 6 years ago
NOTICE modified: .gitignore 6 years ago
README.rst readme rst 4 years ago
composer.json modified: .gitignore 6 years ago
requirements.txt text writer 4 years ago
run.py docstring and comments for code 4 years ago
setup.py readme rst 4 years ago
sitemap.xml aiohttp and aiofile 4 years ago
version.py readme rst 4 years ago

README.rst

pysitemap
=========

Sitemap generator

installing
----------

::

pip install sitemap-generator

requirements
------------

::

asyncio
aiofile
aiohttp

example
-------

::

import sys
import logging
from pysitemap import crawler

if __name__ == '__main__':
if '--iocp' in sys.argv:
from asyncio import events, windows_events
sys.argv.remove('--iocp')
logging.info('using iocp')
el = windows_events.ProactorEventLoop()
events.set_event_loop(el)

# root_url = sys.argv[1]
root_url = 'https://www.haikson.com'
crawler(root_url, out_file='sitemap.xml')

TODO
-----

- big sites with count of pages more then 100K will use more then 100MB
memory. Move queue and done lists into database. Write Queue and Done
backend classes based on
- Lists
- SQLite database
- Redis
- Write api for extending by user backends

changelog
---------

v. 0.9.1
''''''''

- extended readme
- docstrings and code commentaries

v. 0.9.0
''''''''

- since this version package supports only python version >=3.7
- all functions recreated but api saved. If You use this package, then
just update it, install requirements and run process
- all requests works asynchronously