2025-06-19 08:00:18 [scrapy.utils.log] INFO: Scrapy 2.13.2 started (bot: scrapybot) 2025-06-19 08:00:18 [scrapy.utils.log] INFO: Versions: {'lxml': '5.4.0', 'libxml2': '2.13.8', 'cssselect': '1.3.0', 'parsel': '1.10.0', 'w3lib': '2.3.1', 'Twisted': '25.5.0', 'Python': '3.10.18 (main, Jun 10 2025, 23:52:59) [GCC 12.2.0]', 'pyOpenSSL': '25.1.0 (OpenSSL 3.5.0 8 Apr 2025)', 'cryptography': '45.0.4', 'Platform': 'Linux-6.8.0-55-generic-x86_64-with-glibc2.36'} 2025-06-19 08:00:18 [scrapy.addons] INFO: Enabled addons: [] 2025-06-19 08:00:18 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor 2025-06-19 08:00:18 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.throttle.AutoThrottle'] 2025-06-19 08:00:18 [scrapy.crawler] INFO: Overridden settings: {'AUTOTHROTTLE_DEBUG': True, 'AUTOTHROTTLE_ENABLED': True, 'AUTOTHROTTLE_TARGET_CONCURRENCY': 4.0, 'COMMANDS_MODULE': 'houses.commands', 'CONCURRENT_REQUESTS': 8, 'DEFAULT_ITEM_CLASS': 'houses.items.HousesItem', 'DOWNLOAD_DELAY': 0.3, 'FTP_PASSWORD': 'H4KgYJ4zNyTTVokQGa_4', 'FTP_USER': 'ftpuser', 'HTTPCACHE_DIR': '/etc/scrapyd_cache', 'HTTPCACHE_ENABLED': True, 'HTTPCACHE_POLICY': 'crawler_utils.httpcache.GradualExpiresPolicy', 'HTTPCACHE_STORAGE': 'crawler_utils.httpcache.FilesystemCacheStorage', 'LOG_FILE': '/app/logs/rental_nl/valeriusrentals/6859a6ca4ce311f089dfae2447f2c79f.log', 'NEWSPIDER_MODULE': 'houses.spiders', 'RANDOMIZE_DOWNLOAD_DELAY': False, 'SPIDER_MODULES': ['houses.spiders'], 'TELNETCONSOLE_ENABLED': False, 'TWISTED_REACTOR': 'twisted.internet.epollreactor.EPollReactor', 'URLLENGTH_LIMIT': 5000, 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, ' 'like Gecko) Chrome/29.0.1547.65 Safari/537.36'} 2025-06-19 08:00:18 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'houses.middlewares.Fake404Middleware', 'crawler_utils.middlewares.MyUserAgentMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'rotating_proxies.middlewares.RotatingProxyMiddleware', 'rotating_proxies.middlewares.BanDetectionMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats', 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware'] 2025-06-19 08:00:18 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.start.StartSpiderMiddleware', 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2025-06-19 08:00:18 [crawler_utils.yaml_export_pipeline] INFO: YamlExportPipeline initialized with FTP host: crawler-data.boligzonen.dk, user: ftpuser 2025-06-19 08:00:18 [crawler_utils.yaml_export_pipeline] INFO: Output directories: ['upload/rental_nl'] 2025-06-19 08:00:18 [crawler_utils.yaml_export_pipeline] INFO: Output tempdir: /tmp/rental_nl 2025-06-19 08:00:18 [crawler_utils.yaml_export_pipeline] INFO: Using PostCodeValidator for project: rental_nl 2025-06-19 08:00:18 [scrapy.middleware] INFO: Enabled item pipelines: ['houses.pipelines.DuplicatesPipeline', 'crawler_utils.yaml_export_pipeline.YamlExportPipeline'] 2025-06-19 08:00:18 [scrapy.core.engine] INFO: Spider opened 2025-06-19 08:00:18 [py.warnings] WARNING: /usr/local/lib/python3.10/site-packages/scrapy/core/spidermw.py:433: ScrapyDeprecationWarning: houses.spiders.valeriusrentals.ValeriusrentalsSpider defines the deprecated start_requests() method. start_requests() has been deprecated in favor of a new method, start(), to support asynchronous code execution. start_requests() will stop being called in a future version of Scrapy. If you use Scrapy 2.13 or higher only, replace start_requests() with start(); note that start() is a coroutine (async def). If you need to maintain compatibility with lower Scrapy versions, when overriding start_requests() in a spider class, override start() as well; you can use super() to reuse the inherited start() implementation without copy-pasting. See the release notes of Scrapy 2.13 for details: https://docs.scrapy.org/en/2.13/news.html warn( 2025-06-19 08:00:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2025-06-19 08:00:18 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 0, unchecked: 7, reanimated: 0, mean backoff time: 0s) 2025-06-19 08:00:18 [scrapy.extensions.throttle] INFO: slot: ch.proxymesh.com | conc: 1 | delay: 2565 ms (-2434) | latency: 522 ms | size: 29968 bytes 2025-06-19 08:00:18 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:18 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-19 08:00:18 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2025-06-19 08:00:21 [scrapy.extensions.throttle] INFO: slot: jp.proxymesh.com | conc: 1 | delay: 2753 ms (-2246) | latency: 2031 ms | size: 29734 bytes 2025-06-19 08:00:21 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:21 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-19 08:00:21 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.valeriusrentals.com/rentals/?limit=60&wplpage=1) 2025-06-19 08:00:21 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.valeriusrentals.com/for-rent/841-Apartment-Visseringstraat-21-D-Amsterdam/> {'area': 116, 'available': '2025-06-19', 'balcony': True, 'description': '.\n' 'Property Description.\n' 'Spacious and well laid out 2 bedroom apartment in the quiet ' 'part of the Westerpark neighbourhood..\n' 'The apartment is available for rent both FURNISHED and ' 'UNFURNISHED and comes with 1 private parking spot..\n' 'The apartment is very centrally located. Within cycling ' 'distance you will find the Jordaan and the centre of ' 'Amsterdam on one side and the Westerpark with culture park ' 'Westergasfabriek and the trendy Baarsjes on the other. Within ' 'walking distance are several shops as well as a wide range of ' 'restaurants..\n' 'INDELING.\n' 'Through the communal entrance of the complex, you can take ' 'the lift as well as the stairs to the third floor, where the ' 'apartment is located. You enter the apartment in the hall, ' 'equipped with wardrobe and meter cupboard. The hall provides ' 'access to all areas of the flat..\n' 'The spacious living room with open kitchen is very ' 'practically laid out. There is room for both a lounge and a ' 'dining area. The open kitchen is located at the rear of the ' 'apartment and is equipped with an oven and microwave ' 'combination, a gas cooker, extractor hood, dishwasher and ' 'fridge freezer. A balcony is located on either side of the ' 'living room and kitchen, facing west and east respectively..\n' 'In the kitchen there is another room where the washing ' 'machine and dryer are located..\n' 'The two bedrooms are of good size and can accommodate both a ' 'double bed and a good wardrobe. The room at the rear of the ' 'flat has a fixed built-in wardrobe..\n' 'Both the bathroom and the separate toilet are located in the ' 'middle of the flat. The modern bathroom is equipped with a ' 'wide walk-in shower with rain and hand shower, a Jacuzzi and ' 'a wide washbasin with two sinks and built-in taps..\n' 'The entire apartment has a natural-coloured tiled floor also ' 'features the advanced Loxone home automation system..\n' 'In the basement is the parking garage located as well as the ' 'separate storage room..\n' 'There is a large communal roof terrace that can be used by ' 'all residents of the complex..\n' 'LOCATION.\n' 'The beautiful complex is located in a special part of ' 'Amsterdam Westerpark and has views over the water. The ' 'complex is located on a car-free street. In the immediate ' 'vicinity are numerous shops for daily groceries as well as ' 'various specialty shops. Via the Buysbrug, the Hugo de ' 'Grootplein leads directly into the bustling city centre and ' 'the Jordaan, surrounded by special galleries, specialty shops ' 'and quirky boutiques. Via Bilderdijkstraat or Jan van ' 'Galenstraat, you enter the West district. This neighbourhood ' 'is characterised as hip, mixed, cosy and full of trendy cafés ' 'and restaurants. The popular Hallen are a city landmark. ' 'Public transport is very nearby and the Ring A-10 can be ' 'reached in a short time..\n' 'Rent is excluding g/w/e/ tv and internet and local taxes..\n' 'DISCLAIMER.\n' 'This information has been compiled by us with the necessary ' 'care. On our part, however, no liability is accepted for any ' 'incompleteness, inaccuracy or otherwise, or its consequences. ' 'All sizes and dimensions are indicative..\n' ' .', 'energy_label': 'a_2015', 'external_property_attributes': {'external_property_id': '1910f7dbdfad4ecffacfec5df193f6bf', 'external_provider_id': 837, 'property_url': 'https://www.valeriusrentals.com/for-rent/841-Apartment-Visseringstraat-21-D-Amsterdam/'}, 'floor': 3, 'house_number': '21D', 'images': ['https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_01-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_02-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_03-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_04-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_05-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_06-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_07-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_08-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_09-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_10-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_11-2-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_12-2-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_13-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_14-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_15-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_16-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_17-2-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_18-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_19-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_20-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_21-2-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_22-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_23-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_24-1-.jpg', 'https://www.valeriusrentals.com/wp-content/uploads/WPL/841/img_25-2-.jpg'], 'property_type': 'Apartment', 'rent': 3350, 'rooms': 2, 'street_name': 'Visseringstraat', 'zip_code_code': '1051 KH'} 2025-06-19 08:00:21 [scrapy.core.engine] INFO: Closing spider (finished) 2025-06-19 08:00:21 [crawler_utils.yaml_export_pipeline] INFO: Uploading /tmp/rental_nl/valeriusrentals_1750320021.yaml to /upload/rental_nl 2025-06-19 08:00:21 [crawler_utils.yaml_export_pipeline] INFO: FTP host: crawler-data.boligzonen.dk, user: ftpuser 2025-06-19 08:00:21 [paramiko.transport] DEBUG: starting thread (client mode): 0x2c48ac0 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Local version/idstring: SSH-2.0-paramiko_3.5.1 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.11 2025-06-19 08:00:21 [paramiko.transport] INFO: Connected (version 2.0, client OpenSSH_9.6p1) 2025-06-19 08:00:21 [paramiko.transport] DEBUG: === Key exchange possibilities === 2025-06-19 08:00:21 [paramiko.transport] DEBUG: kex algos: sntrup761x25519-sha512@openssh.com, curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group14-sha256, ext-info-s, kex-strict-s-v00@openssh.com 2025-06-19 08:00:21 [paramiko.transport] DEBUG: server key: rsa-sha2-512, rsa-sha2-256, ecdsa-sha2-nistp256, ssh-ed25519 2025-06-19 08:00:21 [paramiko.transport] DEBUG: client encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com 2025-06-19 08:00:21 [paramiko.transport] DEBUG: server encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com 2025-06-19 08:00:21 [paramiko.transport] DEBUG: client mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1 2025-06-19 08:00:21 [paramiko.transport] DEBUG: server mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1 2025-06-19 08:00:21 [paramiko.transport] DEBUG: client compress: none, zlib@openssh.com 2025-06-19 08:00:21 [paramiko.transport] DEBUG: server compress: none, zlib@openssh.com 2025-06-19 08:00:21 [paramiko.transport] DEBUG: client lang: 2025-06-19 08:00:21 [paramiko.transport] DEBUG: server lang: 2025-06-19 08:00:21 [paramiko.transport] DEBUG: kex follows: False 2025-06-19 08:00:21 [paramiko.transport] DEBUG: === Key exchange agreements === 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Strict kex mode: True 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Kex: curve25519-sha256@libssh.org 2025-06-19 08:00:21 [paramiko.transport] DEBUG: HostKey: ssh-ed25519 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Cipher: aes128-ctr 2025-06-19 08:00:21 [paramiko.transport] DEBUG: MAC: hmac-sha2-256 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Compression: none 2025-06-19 08:00:21 [paramiko.transport] DEBUG: === End of kex handshake === 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Resetting outbound seqno after NEWKEYS due to strict mode 2025-06-19 08:00:21 [paramiko.transport] DEBUG: kex engine KexCurve25519 specified hash_algo 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Switch to new keys ... 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Resetting inbound seqno after NEWKEYS due to strict mode 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Got EXT_INFO: {'server-sig-algs': b'ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256', 'publickey-hostbound@openssh.com': b'0', 'ping@openssh.com': b'0'} 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Attempting password auth... 2025-06-19 08:00:21 [paramiko.transport] DEBUG: userauth is OK 2025-06-19 08:00:21 [paramiko.transport] INFO: Authentication (password) successful! 2025-06-19 08:00:21 [paramiko.transport] DEBUG: [chan 0] Max packet in: 32768 bytes 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Received global request "hostkeys-00@openssh.com" 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Rejecting "hostkeys-00@openssh.com" global request from server. 2025-06-19 08:00:21 [paramiko.transport] DEBUG: [chan 0] Max packet out: 32768 bytes 2025-06-19 08:00:21 [paramiko.transport] DEBUG: Secsh channel 0 opened. 2025-06-19 08:00:21 [paramiko.transport] DEBUG: [chan 0] Sesch channel 0 request ok 2025-06-19 08:00:21 [paramiko.transport.sftp] INFO: [chan 0] Opened sftp connection (server version 3) 2025-06-19 08:00:21 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_nl/valeriusrentals_1750320021.yaml', 'wb') 2025-06-19 08:00:21 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_nl/valeriusrentals_1750320021.yaml', 'wb') -> 00000000 2025-06-19 08:00:21 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000) 2025-06-19 08:00:21 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/rental_nl/valeriusrentals_1750320021.yaml') 2025-06-19 08:00:21 [paramiko.transport.sftp] INFO: [chan 0] sftp session closed. 2025-06-19 08:00:21 [paramiko.transport] DEBUG: [chan 0] EOF sent (0) 2025-06-19 08:00:21 [crawler_utils.yaml_export_pipeline] INFO: Successfully uploaded /tmp/rental_nl/valeriusrentals_1750320021.yaml to upload/rental_nl 2025-06-19 08:00:21 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 780, 'downloader/request_count': 2, 'downloader/request_method_count/GET': 2, 'downloader/response_bytes': 61662, 'downloader/response_count': 2, 'downloader/response_status_count/200': 2, 'elapsed_time_seconds': 3.15459, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2025, 6, 19, 8, 0, 21, 466065, tzinfo=datetime.timezone.utc), 'httpcache/firsthand': 2, 'httpcache/miss': 2, 'httpcache/uncacheable': 2, 'httpcompression/response_bytes': 293506, 'httpcompression/response_count': 2, 'item_scraped_count': 1, 'items_per_minute': 20.0, 'log_count/DEBUG': 47, 'log_count/INFO': 24, 'log_count/WARNING': 1, 'memusage/max': 96935936, 'memusage/startup': 96935936, 'proxies/good': 2, 'proxies/mean_backoff': 0.0, 'proxies/reanimated': 0, 'proxies/unchecked': 6, 'request_depth_max': 1, 'response_received_count': 2, 'responses_per_minute': 40.0, 'scheduler/dequeued': 2, 'scheduler/dequeued/memory': 2, 'scheduler/enqueued': 2, 'scheduler/enqueued/memory': 2, 'start_time': datetime.datetime(2025, 6, 19, 8, 0, 18, 311475, tzinfo=datetime.timezone.utc)} 2025-06-19 08:00:21 [scrapy.core.engine] INFO: Spider closed (finished) 2025-06-19 08:00:21 [paramiko.transport] DEBUG: EOF in transport thread