2025-06-19 08:00:10 [scrapy.utils.log] INFO: Scrapy 2.13.2 started (bot: scrapybot) 2025-06-19 08:00:10 [scrapy.utils.log] INFO: Versions: {'lxml': '5.4.0', 'libxml2': '2.13.8', 'cssselect': '1.3.0', 'parsel': '1.10.0', 'w3lib': '2.3.1', 'Twisted': '25.5.0', 'Python': '3.10.18 (main, Jun 10 2025, 23:52:59) [GCC 12.2.0]', 'pyOpenSSL': '25.1.0 (OpenSSL 3.5.0 8 Apr 2025)', 'cryptography': '45.0.4', 'Platform': 'Linux-6.8.0-55-generic-x86_64-with-glibc2.36'} 2025-06-19 08:00:10 [scrapy.addons] INFO: Enabled addons: [] 2025-06-19 08:00:10 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor 2025-06-19 08:00:10 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.throttle.AutoThrottle'] 2025-06-19 08:00:10 [scrapy.crawler] INFO: Overridden settings: {'AUTOTHROTTLE_DEBUG': True, 'AUTOTHROTTLE_ENABLED': True, 'AUTOTHROTTLE_TARGET_CONCURRENCY': 4.0, 'COMMANDS_MODULE': 'houses.commands', 'CONCURRENT_REQUESTS': 8, 'DEFAULT_ITEM_CLASS': 'houses.items.HousesItem', 'DOWNLOAD_DELAY': 0.3, 'FTP_PASSWORD': 'H4KgYJ4zNyTTVokQGa_4', 'FTP_USER': 'ftpuser', 'HTTPCACHE_DIR': '/etc/scrapyd_cache', 'HTTPCACHE_ENABLED': True, 'HTTPCACHE_POLICY': 'crawler_utils.httpcache.GradualExpiresPolicy', 'HTTPCACHE_STORAGE': 'crawler_utils.httpcache.FilesystemCacheStorage', 'LOG_FILE': '/app/logs/rental_nl/compasshousingamsterdam/676b8d464ce311f089dfae2447f2c79f.log', 'NEWSPIDER_MODULE': 'houses.spiders', 'RANDOMIZE_DOWNLOAD_DELAY': False, 'SPIDER_MODULES': ['houses.spiders'], 'TELNETCONSOLE_ENABLED': False, 'TWISTED_REACTOR': 'twisted.internet.epollreactor.EPollReactor', 'URLLENGTH_LIMIT': 5000, 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, ' 'like Gecko) Chrome/29.0.1547.65 Safari/537.36'} 2025-06-19 08:00:11 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'houses.middlewares.Fake404Middleware', 'crawler_utils.middlewares.MyUserAgentMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'rotating_proxies.middlewares.RotatingProxyMiddleware', 'rotating_proxies.middlewares.BanDetectionMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats', 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware'] 2025-06-19 08:00:11 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.start.StartSpiderMiddleware', 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2025-06-19 08:00:11 [crawler_utils.yaml_export_pipeline] INFO: YamlExportPipeline initialized with FTP host: crawler-data.boligzonen.dk, user: ftpuser 2025-06-19 08:00:11 [crawler_utils.yaml_export_pipeline] INFO: Output directories: ['upload/rental_nl'] 2025-06-19 08:00:11 [crawler_utils.yaml_export_pipeline] INFO: Output tempdir: /tmp/rental_nl 2025-06-19 08:00:11 [crawler_utils.yaml_export_pipeline] INFO: Using PostCodeValidator for project: rental_nl 2025-06-19 08:00:11 [scrapy.middleware] INFO: Enabled item pipelines: ['houses.pipelines.DuplicatesPipeline', 'crawler_utils.yaml_export_pipeline.YamlExportPipeline'] 2025-06-19 08:00:11 [scrapy.core.engine] INFO: Spider opened 2025-06-19 08:00:11 [py.warnings] WARNING: /usr/local/lib/python3.10/site-packages/scrapy/core/spidermw.py:433: ScrapyDeprecationWarning: houses.spiders.compasshousingamsterdam.CompasshousingamsterdamSpider defines the deprecated start_requests() method. start_requests() has been deprecated in favor of a new method, start(), to support asynchronous code execution. start_requests() will stop being called in a future version of Scrapy. If you use Scrapy 2.13 or higher only, replace start_requests() with start(); note that start() is a coroutine (async def). If you need to maintain compatibility with lower Scrapy versions, when overriding start_requests() in a spider class, override start() as well; you can use super() to reuse the inherited start() implementation without copy-pasting. See the release notes of Scrapy 2.13 for details: https://docs.scrapy.org/en/2.13/news.html warn( 2025-06-19 08:00:11 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2025-06-19 08:00:11 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 0, unchecked: 7, reanimated: 0, mean backoff time: 0s) 2025-06-19 08:00:11 [scrapy.extensions.throttle] INFO: slot: nl.proxymesh.com | conc: 1 | delay: 2567 ms (-2432) | latency: 542 ms | size: 24656 bytes 2025-06-19 08:00:11 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:11 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-19 08:00:11 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2025-06-19 08:00:12 [scrapy.extensions.throttle] INFO: slot: de.proxymesh.com | conc: 1 | delay: 2571 ms (-2428) | latency: 570 ms | size: 14556 bytes 2025-06-19 08:00:12 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:12 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-19 08:00:12 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:12 [scrapy.extensions.throttle] INFO: slot: ch.proxymesh.com | conc: 1 | delay: 2573 ms (-2426) | latency: 587 ms | size: 16309 bytes 2025-06-19 08:00:12 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:12 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-19 08:00:12 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:12 [scrapy.extensions.throttle] INFO: slot: fr.proxymesh.com | conc: 1 | delay: 2581 ms (-2418) | latency: 648 ms | size: 17222 bytes 2025-06-19 08:00:12 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:12 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-19 08:00:12 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:13 [scrapy.extensions.throttle] INFO: slot: world.proxymesh.com | conc: 1 | delay: 2645 ms (-2354) | latency: 1162 ms | size: 17010 bytes 2025-06-19 08:00:13 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:13 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-19 08:00:13 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:15 [scrapy.extensions.throttle] INFO: slot: jp.proxymesh.com | conc: 1 | delay: 2893 ms (-2106) | latency: 3149 ms | size: 17642 bytes 2025-06-19 08:00:15 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:15 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-19 08:00:15 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:16 [scrapy.extensions.throttle] INFO: slot: nl.proxymesh.com | conc: 1 | delay: 1351 ms (-1216) | latency: 539 ms | size: 20403 bytes 2025-06-19 08:00:16 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:16 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:16 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.compasshousingamsterdam.com/residential-listings/rent/amsterdam/plantage-muidergracht/ref-1038?moveunavailablelistingstothebottom=true&orderby=10&orderdescending=true&take=50> {'area': 81, 'available': None, 'balcony': False, 'deposit_numb': 6300, 'description': '.\n' 'Description.\n' 'Newly Renovated Apartment in the Elegant Plantage District in ' 'the Heart of Amsterdam!.\n' 'This apartment comes fully furnished and features 2 bedrooms. ' 'It’s a spacious home of approximately 81m², beautifully ' 'styled and furnished. The apartment was completely renovated ' 'this year and is located on the fourth floor of a stunning, ' 'stately building..\n' 'Via a shared staircase (used by only two neighbors), you ' 'enter the apartment on the third floor, followed by a private ' 'entrance to the fourth floor. The entrance and living room ' 'are separated by a small hallway where coats can be hung and ' 'shoes can be removed..\n' 'What immediately stands out is the spacious living room, ' 'modern kitchen, and tastefully decorated dining area..\n' 'The living room is very contemporary, with large rear-facing ' 'windows and authentic wooden ceiling beams that give the home ' 'a warm and welcoming atmosphere..\n' 'The entire apartment features a new (vinyl) herringbone ' 'floor, underfloor heating, and freshly plastered and painted ' 'walls..\n' 'At the center of the living-kitchen area is a brand-new ' 'kitchen that looks very spacious. The kitchen includes ' 'built-in appliances and a countertop nearly 5 meters wide. ' 'Opposite it is a lovely dining table, perfect for enjoying ' 'cozy dinners with family and friends. Black ceiling ' 'spotlights can be adjusted to shine in any direction..\n' 'The bedrooms are located on the street side of the apartment, ' "but it's still remarkably quiet. The larger bedroom can be " 'decorated according to the new tenant’s preference. The ' 'smaller bedroom is furnished with a double bed and ' 'wardrobe..\n' 'Highlights of the property:.\n' '- Energy label A+.\n' '- High-quality finishes.\n' '- Two bedrooms.\n' '- Bathroom with separate toilet and hand bidet.\n' '- Excellent location in Amsterdam, within walking distance of ' 'Artis Zoo, trendy restaurants, and the city center.\n' 'Furthermore, the following applies:.\n' '- Rent excludes utilities (gas, water, electricity) but ' 'connection is shared with owner who lives on third floor. ' 'There are submeters present..\n' '- The apartment does not have its own address; registration ' 'is possible via the landlord’s address..\n' '- Washer/dryer combination included.\n' '- Internet is paid for and shared with the landlord.\n' '- Tenant is not eligible to apply for a parking permit.\n' '- The apartment is rented under a Type A contract.\n' '- A minimum stay of 1 year is required.\n' '- Shared housing (two people) is negotiable.\n' 'The Plantage district is a unique and beautiful part of ' 'Amsterdam, well-known for the Artis Zoo, its wide streets, ' 'and stunning buildings. It’s a quieter neighborhood, yet the ' 'bustling city center is within walking distance. Nearby, ' 'you’ll find many trendy restaurants and bars like De ' 'Juwelier, De Plantage, De Zoldering, Entrepot, and Coulisse. ' 'Other neighborhood highlights include Hortus Botanicus, the ' 'Music Theatre, and Wertheim Park..\n' 'The apartment is conveniently located for public transport. ' 'Tram line 14 is just steps away, and the Waterlooplein metro ' 'station is also within walking distance. The A10 ring road is ' 'quickly accessible via the IJ tunnel!.\n' 'Renting with Compass Housing.\n' '- No commission costs!.\n' '- In-house know-how of local laws and regulations.\n' '- One point of contact.\n' '- Free assistance with setting up your utility ' 'subscriptions.\n' 'Compass Housing – Let us guide you.', 'energy_label': 'a_2020', 'external_property_attributes': {'external_property_id': '1e989a85f7d14f1f7bd167d5780b7768', 'external_provider_id': 848, 'property_url': 'https://www.compasshousingamsterdam.com/residential-listings/rent/amsterdam/plantage-muidergracht/ref-1038?moveunavailablelistingstothebottom=true&orderby=10&orderdescending=true&take=50'}, 'floor': 4, 'furnished': True, 'house_number': '', 'images': ['https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117631/$xl/12.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117632/$xl/14.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117633/$xl/2.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117635/$xl/4.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117634/$xl/1.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117636/$xl/6.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117637/$xl/18.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117638/$xl/7.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117639/$xl/13.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117640/$xl/19.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117641/$xl/20.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117642/$xl/21.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117644/$xl/22.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117643/$xl/23.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117645/$xl/40.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117646/$xl/38.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117647/$xl/44.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117649/$xl/32.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117650/$xl/43.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117648/$xl/31.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117651/$xl/27.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117652/$xl/26.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117653/$xl/28.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117654/$xl/29.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117655/$xl/30.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117656/$xl/47.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117657/$xl/49.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117658/$xl/50.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117659/$xl/51.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117660/$xl/52.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117661/$xl/53.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117662/$xl/53_1.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117663/$xl/54.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117664/$xl/55.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117665/$xl/56.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117668/$xl/57.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3666649/112117670/$xl/58.jpg'], 'property_type': 'Apartment', 'rent': 3250, 'rooms': 3, 'street_name': 'Plantage Muidergracht', 'zip_code_code': '1018 TN'} 2025-06-19 08:00:17 [scrapy.extensions.throttle] INFO: slot: ch.proxymesh.com | conc: 1 | delay: 1314 ms (-1259) | latency: 219 ms | size: 18889 bytes 2025-06-19 08:00:17 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:17 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:17 [scrapy.extensions.throttle] INFO: slot: fr.proxymesh.com | conc: 1 | delay: 1323 ms (-1257) | latency: 265 ms | size: 15443 bytes 2025-06-19 08:00:17 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:17 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:17 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.compasshousingamsterdam.com/residential-listings/rent/amsterdam/cas-oorthuyskade/ref-1039?moveunavailablelistingstothebottom=true&orderby=10&orderdescending=true&take=50> {'area': 85, 'available': None, 'balcony': True, 'deposit_numb': 5000, 'description': '.\n' 'Description.\n' 'Beautiful and very well finished two bedroom apartment ' '(unfurnished) of approximately 85 m² with fantastic views ' 'over ‘het IJ’ lake. The location on IJburg is ideal where you ' 'end up in a wonderfully peaceful living environment, but all ' 'the amenities of the big city are within reach..\n' 'The apartment is located on the sixth floor of a modern ' 'residential complex on the Cas Oorthuyskade with spectacular ' 'views over the IJ lake towards the Durgerdammerdijk and ' 'Zeeburg..\n' 'The property is well maintained and you have a spacious ' 'living room, a luxury open (Siemens) kitchen with cooking ' 'island, two bedrooms and a modern bathroom with a shower and ' 'double sink..\n' 'Layout:.\n' 'Common entrance with mailboxes, doorbells, elevator and ' 'staircase..\n' 'Apartment entrance, hall, spacious living room, open kitchen ' 'with cooking island with appliances, two spacious bedrooms, ' 'modern bathroom with shower and double sink, separate toilet ' 'with hand basin..\n' 'Surroundings:.\n' 'IJburg is characterized by a well-arranged structure, the ' 'water-rich character and the favorable location in relation ' 'to the facilities..\n' 'Within walking distance of the apartment is the shopping ' 'center of IJburg with a diverse range of shops, a primary ' 'school and the port of IJburg. The park, high school, health ' 'center, city beach Blijburg, and various restaurants are also ' 'nearby. With less than 5 minutes driving you are on the A10 ' 'ring road but also roads such as A1, A9 and A2 are easily ' 'accessible..\n' 'Public transport is well arranged and you can reach the ' 'center within 15 minutes by tram 26 and bus 66 will take you ' 'to Zuidoost..', 'energy_label': 'b', 'external_property_attributes': {'external_property_id': '080a1a312c3f08e2e7d967d450aaa9e9', 'external_provider_id': 848, 'property_url': 'https://www.compasshousingamsterdam.com/residential-listings/rent/amsterdam/cas-oorthuyskade/ref-1039?moveunavailablelistingstothebottom=true&orderby=10&orderdescending=true&take=50'}, 'floor': 6, 'furnished': False, 'house_number': '', 'images': ['https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616928/$xl/52124280.1563367332-452.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616932/$xl/52124280.1563367317-284.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616929/$xl/52124280.1563367321-171.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616930/$xl/52124280.1563367327-724.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616931/$xl/52124280.1563367342-451.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616934/$xl/52124280.1563367346-126.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616933/$xl/52124280.1563367351-667.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616939/$xl/52124280.1563367357-59.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616935/$xl/52124280.1563367362-404.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616936/$xl/52124280.1563367367-303.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616937/$xl/52124280.1563367371-221.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616938/$xl/52124280.1563367392-492.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616940/$xl/52124280.1563367377-655.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616941/$xl/52124280.1563367383-536.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616944/$xl/52124280.1563367399-126.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616942/$xl/52124280.1563367406-526.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616945/$xl/52124280.1563367411-541.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3680118/112616943/$xl/52124280.1563367416-736.jpg'], 'property_type': 'Apartment', 'rent': 2600, 'rooms': 3, 'street_name': 'Cas Oorthuyskade', 'zip_code_code': '1087 DP'} 2025-06-19 08:00:17 [scrapy.extensions.throttle] INFO: slot: world.proxymesh.com | conc: 1 | delay: 1390 ms (-1254) | latency: 544 ms | size: 14896 bytes 2025-06-19 08:00:17 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:17 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:17 [scrapy.extensions.throttle] INFO: slot: de.proxymesh.com | conc: 1 | delay: 1314 ms (-1256) | latency: 229 ms | size: 15834 bytes 2025-06-19 08:00:17 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:17 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:19 [scrapy.extensions.throttle] INFO: slot: nl.proxymesh.com | conc: 1 | delay: 735 ms (-615) | latency: 477 ms | size: 16598 bytes 2025-06-19 08:00:19 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:19 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:20 [scrapy.extensions.throttle] INFO: slot: fr.proxymesh.com | conc: 1 | delay: 700 ms (-622) | latency: 311 ms | size: 19173 bytes 2025-06-19 08:00:20 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:20 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.compasshousingamsterdam.com/residential-listings/rent/amsterdam/minervalaan/ref-1028?moveunavailablelistingstothebottom=true&orderby=10&orderdescending=true&take=50> {'area': 121, 'available': None, 'balcony': True, 'deposit_numb': 7000, 'description': '.\n' 'Description.\n' 'Spacious and Light Apartment of approx. 121m² in the Heart of ' 'Amsterdam South located on the Minervalaan(..\n' 'Located on the beautiful Minervalaan, this charming apartment ' 'offers plenty of space and potential. It is an ideal place ' 'for those seeking a home with character and a well-designed ' 'layout, situated in one of the most sought-after ' 'neighborhoods in Amsterdam South. The apartment features a ' 'large balcony at the rear, perfect for enjoying the outdoors ' 'and the view..\n' 'Location/Accessibility:.\n' 'Minervalaan is known for its stately appearance and green ' 'surroundings, with several Berlage elements adding character ' 'to the street. The apartment is situated between Stadionweg ' 'and the cyclist bridge, within walking distance of station ' 'Zuid/WTC. Here, you live peacefully with views over abundant ' 'greenery, while being close to the vibrant Amsterdam South, ' 'which offers a wide variety of shops, restaurants, and ' 'cultural activities. Beethovenstraat, Stadionweg, and ' 'Cornelis Schuytstraat are nearby, along with popular cafes, ' 'lunchrooms, and restaurants. For sports and relaxation, you ' 'can visit the Vondelpark, Beatrixpark, or Amstelpark, all ' 'just a few minutes away..\n' 'The apartment is excellently accessible by public transport, ' 'with Zuid/WTC station just a short distance away, providing ' 'metro, train, and bus connections to Schiphol and other parts ' 'of the city. For those driving, parking is available through ' 'a permit system, and permits are readily obtainable in the ' 'area..\n' 'Layout:.\n' 'Upon entering the apartment, you enter a small hallway that ' 'provides access to various rooms. From this hall, you can ' 'access a separate toilet and a door leading to the first ' 'small bedroom at the front of the apartment. On the opposite ' 'side of the hall is a spacious room at the front, which ' 'receives a lot of natural light through the large windows..\n' 'From the hallway, you enter the large central hall, which ' 'connects to the other rooms. There are two large bedrooms at ' 'the front of the apartment, one of which is connected to the ' 'other via French doors. This provides an open, airy feel and ' 'allows for a flexible layout of the space..\n' 'The living room and dining room are in an en suite layout, ' 'creating a spacious and inviting area. The dining room is ' 'adjacent to the kitchen, and the kitchen has French doors ' 'leading to the dining room, making the spaces feel connected. ' 'This layout is ideal for both relaxing evenings at home and ' 'hosting guests..\n' 'At the rear of the apartment is a large bedroom that opens ' 'onto the balcony. The spacious bathroom, accessible from both ' 'the bedroom and the hall, is equipped with a bathtub with ' 'shower, a double washbasin, and a toilet. This bathroom ' 'offers all the comfort needed for relaxation and ' 'convenience..\n' 'Balcony:.\n' 'The apartment features a large balcony at the rear, perfect ' 'for outdoor seating and enjoying the peaceful surroundings. ' 'The balcony overlooks the greenery and provides ample light ' 'to the back of the apartment..\n' 'In Summary:.\n' 'This apartment offers the perfect balance of space, light, ' 'and character, with a practical layout and the opportunity to ' 'personalize the space to your liking. Located in one of the ' 'most desirable neighborhoods in Amsterdam South, it not only ' 'offers a fantastic location but also a home that has great ' 'potential for the future. It is an ideal place for those ' 'seeking comfort and style, with everything Amsterdam South ' 'has to offer within reach..', 'energy_label': None, 'external_property_attributes': {'external_property_id': '60e4bf516e17785852016d6f361c92db', 'external_provider_id': 848, 'property_url': 'https://www.compasshousingamsterdam.com/residential-listings/rent/amsterdam/minervalaan/ref-1028?moveunavailablelistingstothebottom=true&orderby=10&orderdescending=true&take=50'}, 'floor': 1, 'furnished': False, 'house_number': '', 'images': ['https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688303/$xl/223-456.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688304/$xl/276_14474.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688307/$xl/foto%2006-02-2025%2012%2034%2043.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688305/$xl/foto%2006-02-2025%2012%2035%2018.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688306/$xl/foto%2006-02-2025%2012%2021%2030.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688308/$xl/foto%2006-02-2025%2012%2018%2053.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688309/$xl/foto%2006-02-2025%2012%2019%2018.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688310/$xl/foto%2006-02-2025%2012%2019%2044.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688311/$xl/foto%2006-02-2025%2012%2020%2054.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688312/$xl/foto%2006-02-2025%2012%2020%2032.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688320/$xl/foto%2006-02-2025%2012%2023%2035.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688314/$xl/foto%2006-02-2025%2012%2024%2028.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688317/$xl/foto%2006-02-2025%2012%2024%2040.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688321/$xl/foto%2006-02-2025%2012%2025%2031.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688323/$xl/foto%2006-02-2025%2012%2025%2037.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688325/$xl/foto%2006-02-2025%2012%2029%2046.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688328/$xl/foto%2006-02-2025%2012%2022%2025.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688330/$xl/287_1449087.jpg', 'https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688332/$xl/280_144567.jpg', 'ENCODED:https://haywebattachments.blob.core.windows.net/public/2778/3599892/109688334/$xl/foto%2006-02-2025%2012%2022%2002.jpg'], 'property_type': 'Apartment', 'rent': 3500, 'rooms': 4, 'street_name': 'Minervalaan', 'zip_code_code': '1077 NR'} 2025-06-19 08:00:20 [scrapy.extensions.throttle] INFO: slot: world.proxymesh.com | conc: 1 | delay: 765 ms (-625) | latency: 558 ms | size: 15280 bytes 2025-06-19 08:00:20 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:20 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:20 [scrapy.extensions.throttle] INFO: slot: de.proxymesh.com | conc: 1 | delay: 687 ms (-627) | latency: 239 ms | size: 14714 bytes 2025-06-19 08:00:20 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:20 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:20 [scrapy.extensions.throttle] INFO: slot: nl.proxymesh.com | conc: 1 | delay: 426 ms (-309) | latency: 468 ms | size: 16261 bytes 2025-06-19 08:00:20 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:20 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:20 [scrapy.extensions.throttle] INFO: slot: ch.proxymesh.com | conc: 1 | delay: 711 ms (-603) | latency: 432 ms | size: 15537 bytes 2025-06-19 08:00:20 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:20 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:21 [scrapy.extensions.throttle] INFO: slot: uk.proxymesh.com | conc: 1 | delay: 2588 ms (-2411) | latency: 709 ms | size: 16175 bytes 2025-06-19 08:00:21 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:21 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-19 08:00:21 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:21 [scrapy.extensions.throttle] INFO: slot: nl.proxymesh.com | conc: 1 | delay: 300 ms (-126) | latency: 280 ms | size: 15865 bytes 2025-06-19 08:00:21 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:21 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:21 [scrapy.extensions.throttle] INFO: slot: fr.proxymesh.com | conc: 1 | delay: 388 ms (-312) | latency: 306 ms | size: 15432 bytes 2025-06-19 08:00:21 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:21 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:21 [scrapy.extensions.throttle] INFO: slot: nl.proxymesh.com | conc: 1 | delay: 300 ms (+0) | latency: 429 ms | size: 16173 bytes 2025-06-19 08:00:21 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:21 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:21 [scrapy.extensions.throttle] INFO: slot: world.proxymesh.com | conc: 1 | delay: 460 ms (-304) | latency: 624 ms | size: 17911 bytes 2025-06-19 08:00:21 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:21 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:22 [scrapy.extensions.throttle] INFO: slot: de.proxymesh.com | conc: 1 | delay: 407 ms (-279) | latency: 515 ms | size: 16777 bytes 2025-06-19 08:00:22 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:22 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:22 [scrapy.extensions.throttle] INFO: slot: fr.proxymesh.com | conc: 1 | delay: 300 ms (-88) | latency: 466 ms | size: 15707 bytes 2025-06-19 08:00:22 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:22 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:22 [scrapy.extensions.throttle] INFO: slot: ch.proxymesh.com | conc: 1 | delay: 412 ms (-298) | latency: 458 ms | size: 15857 bytes 2025-06-19 08:00:22 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-19 08:00:22 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.compasshousingamsterdam.com/residential-listings) 2025-06-19 08:00:22 [scrapy.core.engine] INFO: Closing spider (finished) 2025-06-19 08:00:22 [crawler_utils.yaml_export_pipeline] INFO: Uploading /tmp/rental_nl/compasshousingamsterdam_1750320016.yaml to /upload/rental_nl 2025-06-19 08:00:22 [crawler_utils.yaml_export_pipeline] INFO: FTP host: crawler-data.boligzonen.dk, user: ftpuser 2025-06-19 08:00:22 [paramiko.transport] DEBUG: starting thread (client mode): 0xb7ab280 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Local version/idstring: SSH-2.0-paramiko_3.5.1 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.11 2025-06-19 08:00:22 [paramiko.transport] INFO: Connected (version 2.0, client OpenSSH_9.6p1) 2025-06-19 08:00:22 [paramiko.transport] DEBUG: === Key exchange possibilities === 2025-06-19 08:00:22 [paramiko.transport] DEBUG: kex algos: sntrup761x25519-sha512@openssh.com, curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group14-sha256, ext-info-s, kex-strict-s-v00@openssh.com 2025-06-19 08:00:22 [paramiko.transport] DEBUG: server key: rsa-sha2-512, rsa-sha2-256, ecdsa-sha2-nistp256, ssh-ed25519 2025-06-19 08:00:22 [paramiko.transport] DEBUG: client encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com 2025-06-19 08:00:22 [paramiko.transport] DEBUG: server encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com 2025-06-19 08:00:22 [paramiko.transport] DEBUG: client mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1 2025-06-19 08:00:22 [paramiko.transport] DEBUG: server mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1 2025-06-19 08:00:22 [paramiko.transport] DEBUG: client compress: none, zlib@openssh.com 2025-06-19 08:00:22 [paramiko.transport] DEBUG: server compress: none, zlib@openssh.com 2025-06-19 08:00:22 [paramiko.transport] DEBUG: client lang: 2025-06-19 08:00:22 [paramiko.transport] DEBUG: server lang: 2025-06-19 08:00:22 [paramiko.transport] DEBUG: kex follows: False 2025-06-19 08:00:22 [paramiko.transport] DEBUG: === Key exchange agreements === 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Strict kex mode: True 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Kex: curve25519-sha256@libssh.org 2025-06-19 08:00:22 [paramiko.transport] DEBUG: HostKey: ssh-ed25519 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Cipher: aes128-ctr 2025-06-19 08:00:22 [paramiko.transport] DEBUG: MAC: hmac-sha2-256 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Compression: none 2025-06-19 08:00:22 [paramiko.transport] DEBUG: === End of kex handshake === 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Resetting outbound seqno after NEWKEYS due to strict mode 2025-06-19 08:00:22 [paramiko.transport] DEBUG: kex engine KexCurve25519 specified hash_algo 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Switch to new keys ... 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Resetting inbound seqno after NEWKEYS due to strict mode 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Got EXT_INFO: {'server-sig-algs': b'ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256', 'publickey-hostbound@openssh.com': b'0', 'ping@openssh.com': b'0'} 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Attempting password auth... 2025-06-19 08:00:22 [paramiko.transport] DEBUG: userauth is OK 2025-06-19 08:00:22 [paramiko.transport] INFO: Authentication (password) successful! 2025-06-19 08:00:22 [paramiko.transport] DEBUG: [chan 0] Max packet in: 32768 bytes 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Received global request "hostkeys-00@openssh.com" 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Rejecting "hostkeys-00@openssh.com" global request from server. 2025-06-19 08:00:22 [paramiko.transport] DEBUG: [chan 0] Max packet out: 32768 bytes 2025-06-19 08:00:22 [paramiko.transport] DEBUG: Secsh channel 0 opened. 2025-06-19 08:00:22 [paramiko.transport] DEBUG: [chan 0] Sesch channel 0 request ok 2025-06-19 08:00:22 [paramiko.transport.sftp] INFO: [chan 0] Opened sftp connection (server version 3) 2025-06-19 08:00:22 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_nl/compasshousingamsterdam_1750320016.yaml', 'wb') 2025-06-19 08:00:22 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_nl/compasshousingamsterdam_1750320016.yaml', 'wb') -> 00000000 2025-06-19 08:00:22 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000) 2025-06-19 08:00:22 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/rental_nl/compasshousingamsterdam_1750320016.yaml') 2025-06-19 08:00:22 [paramiko.transport.sftp] INFO: [chan 0] sftp session closed. 2025-06-19 08:00:22 [paramiko.transport] DEBUG: [chan 0] EOF sent (0) 2025-06-19 08:00:22 [crawler_utils.yaml_export_pipeline] INFO: Successfully uploaded /tmp/rental_nl/compasshousingamsterdam_1750320016.yaml to upload/rental_nl 2025-06-19 08:00:22 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 16822, 'downloader/request_count': 25, 'downloader/request_method_count/GET': 25, 'downloader/response_bytes': 432603, 'downloader/response_count': 25, 'downloader/response_status_count/200': 25, 'elapsed_time_seconds': 11.150609, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2025, 6, 19, 8, 0, 22, 447576, tzinfo=datetime.timezone.utc), 'httpcache/firsthand': 25, 'httpcache/miss': 25, 'httpcache/uncacheable': 25, 'httpcompression/response_bytes': 2113376, 'httpcompression/response_count': 25, 'item_scraped_count': 3, 'items_per_minute': 16.363636363636363, 'log_count/DEBUG': 77, 'log_count/INFO': 70, 'log_count/WARNING': 1, 'memusage/max': 96399360, 'memusage/startup': 96399360, 'proxies/good': 7, 'proxies/mean_backoff': 0.0, 'proxies/reanimated': 0, 'proxies/unchecked': 0, 'request_depth_max': 1, 'response_received_count': 25, 'responses_per_minute': 136.36363636363637, 'scheduler/dequeued': 25, 'scheduler/dequeued/memory': 25, 'scheduler/enqueued': 25, 'scheduler/enqueued/memory': 25, 'start_time': datetime.datetime(2025, 6, 19, 8, 0, 11, 296967, tzinfo=datetime.timezone.utc)} 2025-06-19 08:00:22 [scrapy.core.engine] INFO: Spider closed (finished) 2025-06-19 08:00:22 [paramiko.transport] DEBUG: EOF in transport thread