Viewing posts tagged AWS

Adapting Scrapy's images pipeline to use Azure Storage

Although it's not documented, the Scrapy images pipeline supports storing your downloaded images in an Amazon S3 bucket. This is very useful especially when scraping is over distributed nodes and you want a centralized image store.

Read more