2025-06-18 16:00:10 [scrapy.utils.log] INFO: Scrapy 2.13.2 started (bot: scrapybot) 2025-06-18 16:00:10 [scrapy.utils.log] INFO: Versions: {'lxml': '5.4.0', 'libxml2': '2.13.8', 'cssselect': '1.3.0', 'parsel': '1.10.0', 'w3lib': '2.3.1', 'Twisted': '25.5.0', 'Python': '3.10.18 (main, Jun 10 2025, 23:52:59) [GCC 12.2.0]', 'pyOpenSSL': '25.1.0 (OpenSSL 3.5.0 8 Apr 2025)', 'cryptography': '45.0.4', 'Platform': 'Linux-6.8.0-55-generic-x86_64-with-glibc2.36'} 2025-06-18 16:00:10 [scrapy.addons] INFO: Enabled addons: [] 2025-06-18 16:00:10 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor 2025-06-18 16:00:10 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.throttle.AutoThrottle'] 2025-06-18 16:00:10 [scrapy.crawler] INFO: Overridden settings: {'AUTOTHROTTLE_DEBUG': True, 'AUTOTHROTTLE_ENABLED': True, 'AUTOTHROTTLE_TARGET_CONCURRENCY': 2, 'COMMANDS_MODULE': 'houses.commands', 'CONCURRENT_REQUESTS': 2, 'CONCURRENT_REQUESTS_PER_DOMAIN': 2, 'DEFAULT_ITEM_CLASS': 'houses.items.HousesItem', 'DOWNLOAD_DELAY': 0.3, 'FTP_PASSWORD': 'H4KgYJ4zNyTTVokQGa_4', 'FTP_USER': 'ftpuser', 'HTTPCACHE_DIR': '/etc/scrapyd_cache', 'HTTPCACHE_POLICY': 'crawler_utils.httpcache.GradualExpiresPolicy', 'HTTPCACHE_STORAGE': 'crawler_utils.httpcache.FilesystemCacheStorage', 'LOG_FILE': '/app/logs/rental_nl/expatrentals/4bc4b2744c5d11f089dfae2447f2c79f.log', 'NEWSPIDER_MODULE': 'houses.spiders', 'SPIDER_MODULES': ['houses.spiders'], 'TELNETCONSOLE_ENABLED': False, 'TWISTED_REACTOR': 'twisted.internet.epollreactor.EPollReactor', 'URLLENGTH_LIMIT': 5000, 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, ' 'like Gecko) Chrome/29.0.1547.65 Safari/537.36'} 2025-06-18 16:00:11 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'houses.middlewares.Fake404Middleware', 'crawler_utils.middlewares.MyUserAgentMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'rotating_proxies.middlewares.RotatingProxyMiddleware', 'rotating_proxies.middlewares.BanDetectionMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2025-06-18 16:00:11 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.start.StartSpiderMiddleware', 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2025-06-18 16:00:11 [crawler_utils.yaml_export_pipeline] INFO: YamlExportPipeline initialized with FTP host: crawler-data.boligzonen.dk, user: ftpuser 2025-06-18 16:00:11 [crawler_utils.yaml_export_pipeline] INFO: Output directories: ['upload/rental_nl'] 2025-06-18 16:00:11 [crawler_utils.yaml_export_pipeline] INFO: Output tempdir: /tmp/rental_nl 2025-06-18 16:00:11 [crawler_utils.yaml_export_pipeline] INFO: Using PostCodeValidator for project: rental_nl 2025-06-18 16:00:11 [scrapy.middleware] INFO: Enabled item pipelines: ['houses.pipelines.DuplicatesPipeline', 'crawler_utils.yaml_export_pipeline.YamlExportPipeline'] 2025-06-18 16:00:11 [scrapy.core.engine] INFO: Spider opened 2025-06-18 16:00:11 [py.warnings] WARNING: /usr/local/lib/python3.10/site-packages/scrapy/core/spidermw.py:433: ScrapyDeprecationWarning: houses.spiders.expatrentals.ExpatrentalsSpider defines the deprecated start_requests() method. start_requests() has been deprecated in favor of a new method, start(), to support asynchronous code execution. start_requests() will stop being called in a future version of Scrapy. If you use Scrapy 2.13 or higher only, replace start_requests() with start(); note that start() is a coroutine (async def). If you need to maintain compatibility with lower Scrapy versions, when overriding start_requests() in a spider class, override start() as well; you can use super() to reuse the inherited start() implementation without copy-pasting. See the release notes of Scrapy 2.13 for details: https://docs.scrapy.org/en/2.13/news.html warn( 2025-06-18 16:00:11 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2025-06-18 16:00:11 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 0, unchecked: 7, reanimated: 0, mean backoff time: 0s) 2025-06-18 16:00:12 [scrapy.extensions.throttle] INFO: slot: de.proxymesh.com | conc: 1 | delay: 2697 ms (-2302) | latency: 790 ms | size: 34359 bytes 2025-06-18 16:00:12 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-18 16:00:12 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2025-06-18 16:00:14 [scrapy.extensions.throttle] INFO: slot: de.proxymesh.com | conc: 1 | delay: 1442 ms (-1255) | latency: 374 ms | size: 27597 bytes 2025-06-18 16:00:15 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://expatrentals.com/listings/) 2025-06-18 16:00:15 [logger] INFO: Got result for (Saenredamstraat,Amsterdam,Netherlands) from cache 2025-06-18 16:00:15 [scrapy.extensions.throttle] INFO: slot: jp.proxymesh.com | conc: 1 | delay: 3133 ms (-1866) | latency: 2532 ms | size: 27732 bytes 2025-06-18 16:00:15 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-18 16:00:15 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://expatrentals.com/listings/) 2025-06-18 16:00:15 [logger] INFO: Got result for (Johan van der Keukenstraat,Amsterdam,Netherlands) from cache 2025-06-18 16:00:18 [scrapy.extensions.throttle] INFO: slot: world.proxymesh.com | conc: 1 | delay: 3184 ms (-1815) | latency: 2738 ms | size: 27479 bytes 2025-06-18 16:00:18 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-18 16:00:18 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://expatrentals.com/listings/) 2025-06-18 16:00:18 [logger] INFO: Got result for (Swammerdamstraat,Amsterdam,Netherlands) from cache 2025-06-18 16:00:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://expatrentals.com/listings/amsterdam-swammerdamstraat/> {'area': 99, 'balcony': True, 'deposit_numb': 0, 'description': '.\n' '– Amsterdam East.\n' '– 3 bedrooms.\n' '– Unfurnished.\n' '– Balcony.\n' '– Energy label A.\n' '– Lift.\n' '– Recently renovated.\n' '– Sharing possible up to 2 persons maximum.\n' '– No pets!.\n' 'A very nicely, completely renovated 3 bedroom, unfurnished ' 'apartment on a great location!!.\n' 'The apartment is situated on the second floor of a beautiful ' 'apartment building. The original building dates back from ' '1883, but it has been rebuilt completely in 2017 by the same ' 'architect as the famous Carré Theatre. It’s location on the ' '“Gold Coast of East Amsterdam” is excellent, with some nice ' 'small shops and restaurant “Ysbreeker”, The Amstel Hotel and ' 'Oosterpark nearby. The popular area De Pijp is just a short ' 'bike ride away and for your everyday groceries, there are 2 ' 'big supermarkets closeby. At the Hartog bakery you can get ' 'your daily, freshly baked bread. Public transport is also ' 'very well organised in this area, with several tram, bus and ' 'metro lines within walking distance..\n' 'Lay-out:.\n' 'The apartment is accessible by stairs, or by lift from which ' 'you immediately enter in the property. The living room has 3 ' 'large windows, which give a spacious feeling to the apartment ' 'and brings in a lot of natural light. The modern kitchen is ' 'equipped with all modern day appliances. At the back of the ' 'apartment is the very spacious master bedroom and a second ' 'large bedroom., both giving you access to the sunny balcony. ' 'The third bedroom is smaller, but can still fit a double bed. ' 'The bathroom has a shower, bath tub and a double sink. The ' 'toilet is separate and the washer and dryer are nicely placed ' 'in a closet..\n' 'This apartment is perfect for a couple that needs the extra ' 'space for working from home..\n' 'DISCLAIMER.\n' 'This information has been compiled by us with the necessary ' 'care. On our part , however, no liability is accepted for any ' 'incompleteness, inaccuracy or otherwise, or its consequences. ' 'All sizes and dimensions are indicative..', 'elevator': True, 'energy_label': 'a_2015', 'external_property_attributes': {'external_property_id': '1f103b75b9d088eb2f3ee4a8d597184d', 'external_provider_id': 839, 'property_url': 'https://expatrentals.com/listings/amsterdam-swammerdamstraat/'}, 'floor': 2, 'images': ['https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-f7d3e7a8-7314-4e09-bd27-11a0f3640b05-l-20210325_144333-r4ga02nmyat311e6iaz6f7qctkyaxpx2shlqzwq9o0.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-4a828718-9c46-47fd-a0bc-847995fb45a8-l-20210325_144403-r4ga02nmyat311e6iaz6f7qctkyaxpx2shlqzwq9o0.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-148a3c86-a751-4899-8e8d-7fd727476257-l-20210325_144441-r4ga02nmyat311e6iaz6f7qctkyaxpx2shlqzwq9o0.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-44bae078-2542-4848-8f54-a795c41c95e6-l-20210325_144502-r4ga02nmyat311e6iaz6f7qctkyaxpx2shlqzwq9o0.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-cd3a7063-4aae-45fb-967e-c0d5a6be1433-l-20210325_144517-r4ga02nmyat311e6iaz6f7qctkyaxpx2shlqzwq9o0.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-cfb41c3e-c140-4cb6-b38f-ab4f49577413-l-20210325_144531-r4ga02nmyat311e6iaz6f7qctkyaxpx2shlqzwq9o0.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-9e6b7c43-9265-49b9-9895-cdb842455bb0-l-20210325_145456-r4ga02nmyat311e6iaz6f7qctkyaxpx2shlqzwq9o0.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-f1f568b7-0bb8-4ed5-80e0-ee9842e62d44-l-20210325_144556-r4ga03lh54udcnctctdszphteyto5f0t4m98h6ovhs.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-6edfa34c-73a4-4288-8890-a57d7e3b6987-l-20210325_144632-r4ga03lh54udcnctctdszphteyto5f0t4m98h6ovhs.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-a94f8f6f-86b4-4377-8b02-6a29e312c30a-l-20210325_144730-r4ga03lh54udcnctctdszphteyto5f0t4m98h6ovhs.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-3dc3daf9-bee5-4472-9d5d-8c1dc87528fa-l-20210325_144803-r4ga03lh54udcnctctdszphteyto5f0t4m98h6ovhs.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-4365f95e-4949-4d28-9fdf-a249d5b2cd80-l-20210325_145612-r4ga03lh54udcnctctdszphteyto5f0t4m98h6ovhs.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-455aa996-fb31-4360-8054-0fc82ca31c55-l-20210325_145534-r4ga03lh54udcnctctdszphteyto5f0t4m98h6ovhs.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-50e383f7-600b-442d-9ed7-7cb590ec0ead-l-20210325_145629-r4ga03lh54udcnctctdszphteyto5f0t4m98h6ovhs.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-69e4b408-2f29-4d77-aadc-26d9c7dd90a3-l-20210325_145035-r4ga03lh54udcnctctdszphteyto5f0t4m98h6ovhs.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-5b6fbe75-f2ed-4b42-a0f4-40a57ae53f15-l-20210325_145211-r4ga04jbbyvno9bg7bsfk79a0cp1d44jgqwpygnhbk.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-959f579b-b490-4648-97b3-4413096801b5-l-20210325_144941-r4ga04jbbyvno9bg7bsfk79a0cp1d44jgqwpygnhbk.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-d752f150-4b72-49d1-964f-2113072e521a-l-20210325_144959-r4ga04jbbyvno9bg7bsfk79a0cp1d44jgqwpygnhbk.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-2bee391c-0108-42da-917d-50d73265ca56-l-20210325_144822-r4ga04jbbyvno9bg7bsfk79a0cp1d44jgqwpygnhbk.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-63d3c1e4-ffaa-48c8-855b-23b1016cc7a8-l-20210325_144904-r4ga04jbbyvno9bg7bsfk79a0cp1d44jgqwpygnhbk.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-95f6d727-32f6-41c6-bd4e-c40ea7906366-l-20210325_145340-r4ga04jbbyvno9bg7bsfk79a0cp1d44jgqwpygnhbk.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-a7842d71-2ff6-42df-9cdb-0237e624faca-l-20210325_145301-r4ga04jbbyvno9bg7bsfk79a0cp1d44jgqwpygnhbk.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-b3c0c04f-864a-46f8-8926-a98703ef1707-l-img-20210208-wa0003-002-r4ga05h5iswxzva31u724p0qlqkekt89svk7fqm35c.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-3f2860d2-5dee-4abe-93ff-44133943c65a-l-img_5749-r4ga05h5iswxzva31u724p0qlqkekt89svk7fqm35c.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-c2180b16-f02d-4323-9fcc-9c25e54ac2d9-l-floorplan-r4ga05h5iswxzva31u724p0qlqkekt89svk7fqm35c.jpg'], 'pets_allowed': False, 'property_type': 'Apartment', 'rent': 3450, 'rooms': 4, 'sharing_friendly': True, 'street_name': 'Swammerdamstraat', 'zip_code_code': '1091'} 2025-06-18 16:00:18 [scrapy.extensions.throttle] INFO: slot: jp.proxymesh.com | conc: 1 | delay: 1775 ms (-1357) | latency: 837 ms | size: 27449 bytes 2025-06-18 16:00:18 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://expatrentals.com/listings/) 2025-06-18 16:00:18 [logger] INFO: Got result for (Meidoornweg,Badhoevedorp,Netherlands) from cache 2025-06-18 16:00:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://expatrentals.com/listings/badhoevedorp-meidoornweg-2-b/> {'area': 25, 'deposit_numb': 0, 'description': '.\n' '– Short term Rentals (from 6 nights, untill 3 months).\n' '– Fully furnished and serviced.\n' '– Studio Apartments (variety of sizes).\n' '– Amsterdam – Badhoevedorp.\n' '– Modern, luxurious.\n' '– Prices include: water, electricity, gas, internet Wifi, tv, ' 'Houskeeping, cleaning, Laundry facility, gym.\n' 'A unique ‘short-stay’ Housing project, offering 98 serviced ' 'studio Apartments, available per direct and for a flexible ' 'rental period..\n' 'This newly developed apartment Hotel is located in ' 'Badhoevedorp, opposite the Amsterdam Forest, closeby ' 'Amsterdam and Schiphol..\n' 'This apartment is available for a minimum stay of 6 nights, ' 'and includes:.\n' '– High speed Internet Wifi, TV, electricity, water, heating.\n' '– Housekeeping 1 time a week.\n' '– Bed sheets and towels replacement 1 time a week.\n' '– Use of the Laundry facility of the complex.\n' '– Free access to the gym of the complex..\n' '– Free Wifi.\n' '– double bedding.\n' '– bathroom equipped with shower, sink, toilet.\n' '– kitchen equipped with microwave-oven combi, fridge, ' 'watercooker, coffee and tea facility,.\n' 'there is no stove, but tenant can bring their own electric ' 'stove/pit..\n' '– living area equipped with sofa, desk and chair, wardrobe, ' 'bed, flatscreen tv, wall art.\n' '– Climate control.\n' '– Hairdryer, toilet articles.\n' '– Parking available for €3,- per day extra.\n' 'Prices per month (starting from, and subject to availability) ' ':.\n' '25m2 €2.220.\n' '30m2 €2.495.\n' '40m2 €2.795.\n' '60m2 €3.995.\n' 'Week prices per night (starting from, and subject to ' 'availability):.\n' '25m2 €78.-.\n' '30m2 €85,50.\n' '40m2 €102,-.\n' '50m2 €152,- 2 room apartment, suitable for 4 persons.\n' '60m2 €182,-.\n' '70m2 €202,- 2 room apartment, suitable for 4 persons.\n' 'All prices are excluding 6.05% tourist tax..\n' 'Location:.\n' 'The apartment is located a few minutes away from the Tram ' 'Stop Oudenaardeplantsoen in Badhoevedorp. Tram No 2 brings ' 'you in the city centre of Amsterdam in less than 30 minutes. ' 'Daily grocery can be done at a few local supermarkets within ' 'walking distance. To Schiphol airport you reach the studio in ' 'less them 20 minutes with bus 69 or 197. There is free ' 'parking in front of the apartment, and the A4 and A9 highways ' 'are only 2 minutes away from the apartment. And above all, ' 'the Amsterdam forest is just opposite the apartment and is ' 'the largest recreational park in and around Amsterdam!.', 'energy_label': 'there is no known energy label', 'external_property_attributes': {'external_property_id': '5d2b2505b71e4798ea059f27f5e3649f', 'external_provider_id': 839, 'property_url': 'https://expatrentals.com/listings/badhoevedorp-meidoornweg-2-b/'}, 'floor': 1, 'furnished': True, 'images': ['https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-025acc30-1f8a-46b9-83f8-dd828f6a0339-l-51453527.1513951674-902-qkf5j5svvjgxdsi49sjdpxykcplit3cw290ltt8f3k.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-ae2d41fa-e237-4d77-871d-ccc5d5e3fd13-l-51453527.1513951726-106-qkf5j5svvjgxdsi49sjdpxykcplit3cw290ltt8f3k.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-f1f50666-1b6f-4c4d-9d85-d56b252d61bf-l-51453527.1513951737-114-qkf5j5svvjgxdsi49sjdpxykcplit3cw290ltt8f3k.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-33a41e7d-ea68-43b2-814d-d7af0179d1f9-l-51453527.1513951732-296-qkf5j6qq2di7pegr4ay0afq0y3gw0sgmedo3b370xc.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-072fee99-70ce-4390-b29e-4e61d0d41e0a-l-51453527.1513951679-127-qkf5j6qq2di7pegr4ay0afq0y3gw0sgmedo3b370xc.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-114ee5ac-1f64-4b70-9577-1a532a651a3a-l-51453527.1513951686-284-qkf5j6qq2di7pegr4ay0afq0y3gw0sgmedo3b370xc.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-edee1a6e-293d-4bcb-863e-11e27ab5f631-l-51453527.1513951693-919-qkf5j7ok97ji10fdytcmuxhhjhc98hkcqibksd5mr4.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-4c7c4968-166d-4bda-8953-53ddc3a388d3-l-51453527.1513951698-245-qkf5j7ok97ji10fdytcmuxhhjhc98hkcqibksd5mr4.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-72cffd36-dfb3-4c2e-848c-52647bccc22d-l-51453527.1513951702-898-qkf5j7ok97ji10fdytcmuxhhjhc98hkcqibksd5mr4.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-fc8d97eb-375d-4f23-90b3-a4ac4e33b5d2-l-51453527.1513951708-340-qkf5j8meg1kscme0tbr9ff8y4v7mg6o32mz29n48kw.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-98ab2463-31cf-4016-85ee-958d9b826306-l-51453527.1513951717-635-qkf5j8meg1kscme0tbr9ff8y4v7mg6o32mz29n48kw.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-42b881a5-6b9d-4e40-a636-3401651f5a25-l-51453527.1513951742-371-qkf5j8meg1kscme0tbr9ff8y4v7mg6o32mz29n48kw.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-a2e4bf4a-34ae-4f6e-8627-deb8aa080b06-l-51453527.1513698815-16-qkf5j9k8mvm2o8cnnu5vzx0eq92znvrtermjqx2ueo.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-3d8d0590-d50b-41aa-9f13-84a355e2e64e-l-51453527.1513698822-23-qkf5j9k8mvm2o8cnnu5vzx0eq92znvrtermjqx2ueo.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-8dd08f3e-91af-4b67-aa33-c0c40aba8003-l-51453527.1513698818-878-qkf5j9k8mvm2o8cnnu5vzx0eq92znvrtermjqx2ueo.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-58ddb9fc-49ab-432a-b974-c4758f7d6aa6-l-51453527.1513698825-949-qkf5jai2tpnczubaickikervbmycvkvjqwa1871g8g.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-244432ef-4e49-445e-a65f-0c9281f8bc5a-l-51453527.1513698820-173-qkf5jai2tpnczubaickikervbmycvkvjqwa1871g8g.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-b8815bac-df5e-47cd-a92a-1ee65a947a1d-l-51453527.1513698824-191-qkf5jai2tpnczubaickikervbmycvkvjqwa1871g8g.jpg'], 'property_type': 'Apartment', 'rent': 2220, 'rooms': 1, 'street_name': 'Meidoornweg', 'zip_code_code': '1171'} 2025-06-18 16:00:18 [scrapy.extensions.throttle] INFO: slot: nl.proxymesh.com | conc: 1 | delay: 2655 ms (-2344) | latency: 620 ms | size: 26231 bytes 2025-06-18 16:00:18 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-18 16:00:18 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://expatrentals.com/listings/) 2025-06-18 16:00:19 [scrapy.extensions.throttle] INFO: slot: de.proxymesh.com | conc: 1 | delay: 837 ms (-604) | latency: 467 ms | size: 27429 bytes 2025-06-18 16:00:19 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://expatrentals.com/listings/) 2025-06-18 16:00:19 [logger] INFO: Got result for (Singel,Amsterdam,Netherlands) from cache 2025-06-18 16:00:19 [logger] WARNING: [Google::get_zip] Did not get any results for house number: None, street: Singel, city:Amsterdam, country:Netherlands, trying reverse geocode 2025-06-18 16:00:19 [logger] INFO: Got result for (52.37275857573181,4.890958533766268) from cache 2025-06-18 16:00:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://expatrentals.com/listings/amsterdam-singel/> {'area': 120, 'deposit_numb': 5200, 'description': '.\n' '– Center / Canal belt.\n' '– Furnished.\n' '– 2 bedrooms.\n' 'A very nice and bright apartment on the top floor with a ' 'great canal view!!!.\n' 'Lay-out:.\n' 'Entrance, , indoor stairs to open space living, dining area ' 'and kitchen. The living room with wooden beams at the ceiling ' 'and 3 large windows is at the front of the apartment and has ' 'a very nice view on the Singel .At the back of the apartmemnt ' 'is a good size dining area and open kitched equipped with ' 'alll modern day appliances, including a big oven. Stairs to ' 'the top floor. On this floor there are 2 spacious bedrooms, ' 'both with queen size beds. The master bedroom also has a ' 'wardrobe, TV and a desk. The second bedroom has a cabinet and ' 'the washer and dryer are stored away here in a closet. The ' 'bathroom is inbetween the bedrooms and is fitted with a bath, ' 'separate shower, toilet and wash basin..\n' 'Location:.\n' 'The apartment is located in the center of Amsterdam with ' 'shops, restaurants and public transport within walking ' 'distance..\n' 'This apartment is ideal for a single person or a couple..', 'energy_label': 'c', 'external_property_attributes': {'external_property_id': '2a74a3a2ea6fcf3908519c36de4d686c', 'external_provider_id': 839, 'property_url': 'https://expatrentals.com/listings/amsterdam-singel/'}, 'floor': 3, 'furnished': True, 'images': ['https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-614d7c3d-c3b6-48e5-8992-36c653f19a95-l-living-room-1-big-r4q56gzf1uf7xkfqxanckm0vmifahj163a6vwe7k34.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-c4f43cdb-9c0d-4dca-93e7-767ea5fb5995-l-living-room-3--big-r4q56hx98ogi96edrt1z53sc7wanp84wfeuddo65ww.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-302ccc46-938c-440c-ad95-2af7ef4157d8-l-living-room-2-big-r4q56hx98ogi96edrt1z53sc7wanp84wfeuddo65ww.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-2ff902e6-75a7-4a3f-8538-414fe219475e-l-kitchen-1-big-r4q56hx98ogi96edrt1z53sc7wanp84wfeuddo65ww.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-b473e895-f389-40a9-9603-bd995d66ff49-l-kitchen-2-big-r4q56hx98ogi96edrt1z53sc7wanp84wfeuddo65ww.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-8360c4fa-0f44-4ba7-9ee2-411aaa7388ca-l-kitchen-3-big-r4q56hx98ogi96edrt1z53sc7wanp84wfeuddo65ww.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-0f73ac60-be30-4eaf-bebd-32715b430ec8-l-masterbedroom-big-r4q56hx98ogi96edrt1z53sc7wanp84wfeuddo65ww.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-c549971b-a8f1-4f5d-bc01-aa4c47e3e95b-l-masterbedroom--2-big-r4q56hx98ogi96edrt1z53sc7wanp84wfeuddo65ww.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-9cf5aa46-5e83-42c6-a6ee-43855ac51c81-l-bathroom-1-big-r4q56hx98ogi96edrt1z53sc7wanp84wfeuddo65ww.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-70450191-af2a-4b02-a83e-bab28d455ef0-l-bathroom-2-big-r4q56hx98ogi96edrt1z53sc7wanp84wfeuddo65ww.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-22d0db12-e65c-452c-85e4-e23e6115bb9a-l-bedroom-2-big-r4q56iv3fihsksd0mbglpljsta60wx8mrjhuuy4rqo.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-158ade37-654e-4db6-abde-b8a2f100db2c-l-singel25f13-r4q56iv3fihsksd0mbglpljsta60wx8mrjhuuy4rqo.jpg', 'https://expatrentals.com/wp-content/uploads/elementor/thumbs/pub-e54bc969-b7cf-4722-9172-28b717162b50-l-donzi-r4q56iv3fihsksd0mbglpljsta60wx8mrjhuuy4rqo.jpg'], 'property_type': 'Apartment', 'rent': 3500, 'rooms': 3, 'street_name': 'Singel', 'zip_code_code': '1016 DB'} 2025-06-18 16:00:19 [logger] INFO: Got result for (Sloterkade,Amsterdam,Netherlands) from cache 2025-06-18 16:00:19 [logger] WARNING: [Google::get_zip] Did not get any results for house number: None, street: Sloterkade, city:Amsterdam, country:Netherlands, trying reverse geocode 2025-06-18 16:00:19 [logger] INFO: Got result for (52.3532629,4.8527874) from cache 2025-06-18 16:00:19 [scrapy.core.engine] INFO: Closing spider (finished) 2025-06-18 16:00:19 [crawler_utils.yaml_export_pipeline] INFO: Uploading /tmp/rental_nl/expatrentals_1750262418.yaml to /upload/rental_nl 2025-06-18 16:00:19 [crawler_utils.yaml_export_pipeline] INFO: FTP host: crawler-data.boligzonen.dk, user: ftpuser 2025-06-18 16:00:19 [paramiko.transport] DEBUG: starting thread (client mode): 0x277848b0 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Local version/idstring: SSH-2.0-paramiko_3.5.1 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.11 2025-06-18 16:00:19 [paramiko.transport] INFO: Connected (version 2.0, client OpenSSH_9.6p1) 2025-06-18 16:00:19 [paramiko.transport] DEBUG: === Key exchange possibilities === 2025-06-18 16:00:19 [paramiko.transport] DEBUG: kex algos: sntrup761x25519-sha512@openssh.com, curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group14-sha256, ext-info-s, kex-strict-s-v00@openssh.com 2025-06-18 16:00:19 [paramiko.transport] DEBUG: server key: rsa-sha2-512, rsa-sha2-256, ecdsa-sha2-nistp256, ssh-ed25519 2025-06-18 16:00:19 [paramiko.transport] DEBUG: client encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com 2025-06-18 16:00:19 [paramiko.transport] DEBUG: server encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com 2025-06-18 16:00:19 [paramiko.transport] DEBUG: client mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1 2025-06-18 16:00:19 [paramiko.transport] DEBUG: server mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1 2025-06-18 16:00:19 [paramiko.transport] DEBUG: client compress: none, zlib@openssh.com 2025-06-18 16:00:19 [paramiko.transport] DEBUG: server compress: none, zlib@openssh.com 2025-06-18 16:00:19 [paramiko.transport] DEBUG: client lang: 2025-06-18 16:00:19 [paramiko.transport] DEBUG: server lang: 2025-06-18 16:00:19 [paramiko.transport] DEBUG: kex follows: False 2025-06-18 16:00:19 [paramiko.transport] DEBUG: === Key exchange agreements === 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Strict kex mode: True 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Kex: curve25519-sha256@libssh.org 2025-06-18 16:00:19 [paramiko.transport] DEBUG: HostKey: ssh-ed25519 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Cipher: aes128-ctr 2025-06-18 16:00:19 [paramiko.transport] DEBUG: MAC: hmac-sha2-256 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Compression: none 2025-06-18 16:00:19 [paramiko.transport] DEBUG: === End of kex handshake === 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Resetting outbound seqno after NEWKEYS due to strict mode 2025-06-18 16:00:19 [paramiko.transport] DEBUG: kex engine KexCurve25519 specified hash_algo 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Switch to new keys ... 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Resetting inbound seqno after NEWKEYS due to strict mode 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Attempting password auth... 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Got EXT_INFO: {'server-sig-algs': b'ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256', 'publickey-hostbound@openssh.com': b'0', 'ping@openssh.com': b'0'} 2025-06-18 16:00:19 [paramiko.transport] DEBUG: userauth is OK 2025-06-18 16:00:19 [paramiko.transport] INFO: Authentication (password) successful! 2025-06-18 16:00:19 [paramiko.transport] DEBUG: [chan 0] Max packet in: 32768 bytes 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Received global request "hostkeys-00@openssh.com" 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Rejecting "hostkeys-00@openssh.com" global request from server. 2025-06-18 16:00:19 [paramiko.transport] DEBUG: [chan 0] Max packet out: 32768 bytes 2025-06-18 16:00:19 [paramiko.transport] DEBUG: Secsh channel 0 opened. 2025-06-18 16:00:19 [paramiko.transport] DEBUG: [chan 0] Sesch channel 0 request ok 2025-06-18 16:00:19 [paramiko.transport.sftp] INFO: [chan 0] Opened sftp connection (server version 3) 2025-06-18 16:00:19 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_nl/expatrentals_1750262418.yaml', 'wb') 2025-06-18 16:00:19 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_nl/expatrentals_1750262418.yaml', 'wb') -> 00000000 2025-06-18 16:00:19 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000) 2025-06-18 16:00:19 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/rental_nl/expatrentals_1750262418.yaml') 2025-06-18 16:00:19 [paramiko.transport.sftp] INFO: [chan 0] sftp session closed. 2025-06-18 16:00:19 [paramiko.transport] DEBUG: [chan 0] EOF sent (0) 2025-06-18 16:00:19 [crawler_utils.yaml_export_pipeline] INFO: Successfully uploaded /tmp/rental_nl/expatrentals_1750262418.yaml to upload/rental_nl 2025-06-18 16:00:19 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 2463, 'downloader/request_count': 7, 'downloader/request_method_count/GET': 7, 'downloader/response_bytes': 202492, 'downloader/response_count': 7, 'downloader/response_status_count/200': 7, 'elapsed_time_seconds': 7.803959, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2025, 6, 18, 16, 0, 19, 130511, tzinfo=datetime.timezone.utc), 'httpcompression/response_bytes': 989516, 'httpcompression/response_count': 7, 'item_scraped_count': 3, 'items_per_minute': 25.714285714285715, 'log_count/DEBUG': 56, 'log_count/INFO': 35, 'log_count/WARNING': 3, 'memusage/max': 96653312, 'memusage/startup': 96653312, 'proxies/good': 4, 'proxies/mean_backoff': 0.0, 'proxies/reanimated': 0, 'proxies/unchecked': 3, 'request_depth_max': 1, 'response_received_count': 7, 'responses_per_minute': 60.0, 'scheduler/dequeued': 7, 'scheduler/dequeued/memory': 7, 'scheduler/enqueued': 7, 'scheduler/enqueued/memory': 7, 'start_time': datetime.datetime(2025, 6, 18, 16, 0, 11, 326552, tzinfo=datetime.timezone.utc)} 2025-06-18 16:00:19 [scrapy.core.engine] INFO: Spider closed (finished) 2025-06-18 16:00:19 [paramiko.transport] DEBUG: EOF in transport thread