"Side Projects" aka VW-adverts

It’s Tuesday, May 9th, 2017

I’ve been wanting to push my latest side project forward for quite some time. It’s now time to do just that.

I’ve had the idea of developing a Microservices based product which finds “VW adverts” based on keywords and phrases. You register, enter your phrases, and off you go.

The tech stack is Spring Boot, with MongoDB. After some investigation, i’m going to use Python Scrapy for the crawling.


design


Scrapy

virtualenv -p python3 spider

$ scrapy startproject spider
Traceback (most recent call last):
[snip]
    TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,
AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'

So this is the first config error. As I want fully automated infrastructure, this will need to be solved. Enough for today :0/

Monday May 30th, 2017

Fixes are easy when you you know how. To fix the OP_NO_TLSv1_1, I had to re-install OpenSSL, and then downgrade the Python module:

pip install Twisted==16.4.1

Scrapy is now working fine :O)

The next task is to get an xpath so I can start getting some advert data.