Sitemap generator
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
Kamo Petrosyan 3c70e32b10 0.3.7 9 years ago
pysitemap 0.3.7 9 years ago
.gitignore Merge remote-tracking branch 'origin/master' 9 years ago
LICENSE Initial commit 9 years ago
NOTICE Create Notice 9 years ago
README.md Update README.md 9 years ago
composer.json Update composer.json 9 years ago
run.py example script changed 9 years ago
setup.py 0.3.7 9 years ago
sitemap.xml Multiprocessing version 0.3.4 9 years ago

README.md

pysitemap

Sitemap generator

installing

pip install sitemap-generator

Gevent

Sitemap-generator uses gevent to implement multiprocessing. Install gevent:

pip install gevent

example

import pysitemap
if __name__=='__main__':
    url = 'http://www.example.com/' # url from to crawl
    logfile = 'errlog.log' # path to logfile
    oformat = 'xml' # output format
    crawl = pysitemap.Crawler(url=url, logfile=logfile, oformat=oformat)
    crawl.crawl()

multiprocessing example

import pysitemap
if __name__=='__main__':
    url = 'http://www.example.com/' # url from to crawl
    logfile = 'errlog.log' # path to logfile
    oformat = 'xml' # output format
    crawl = pysitemap.Crawler(url=url, logfile=logfile, oformat=oformat)
    crawl.crawl(pool_size=10) # 10 parsing processes