2025-06-18 16:00:10 [scrapy.utils.log] INFO: Scrapy 2.13.2 started (bot: scrapybot) 2025-06-18 16:00:10 [scrapy.utils.log] INFO: Versions: {'lxml': '5.4.0', 'libxml2': '2.13.8', 'cssselect': '1.3.0', 'parsel': '1.10.0', 'w3lib': '2.3.1', 'Twisted': '25.5.0', 'Python': '3.10.18 (main, Jun 10 2025, 23:52:59) [GCC 12.2.0]', 'pyOpenSSL': '25.1.0 (OpenSSL 3.5.0 8 Apr 2025)', 'cryptography': '45.0.4', 'Platform': 'Linux-6.8.0-55-generic-x86_64-with-glibc2.36'} 2025-06-18 16:00:10 [scrapy.addons] INFO: Enabled addons: [] 2025-06-18 16:00:10 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor 2025-06-18 16:00:10 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.throttle.AutoThrottle'] 2025-06-18 16:00:10 [scrapy.crawler] INFO: Overridden settings: {'AUTOTHROTTLE_DEBUG': True, 'AUTOTHROTTLE_ENABLED': True, 'AUTOTHROTTLE_TARGET_CONCURRENCY': 4.0, 'COMMANDS_MODULE': 'houses.commands', 'CONCURRENT_REQUESTS': 8, 'DEFAULT_ITEM_CLASS': 'houses.items.HousesItem', 'DOWNLOAD_DELAY': 0.3, 'FTP_PASSWORD': 'H4KgYJ4zNyTTVokQGa_4', 'FTP_USER': 'ftpuser', 'HTTPCACHE_DIR': '/etc/scrapyd_cache', 'HTTPCACHE_ENABLED': True, 'HTTPCACHE_POLICY': 'crawler_utils.httpcache.GradualExpiresPolicy', 'HTTPCACHE_STORAGE': 'crawler_utils.httpcache.FilesystemCacheStorage', 'LOG_FILE': '/app/logs/rental_nl/listings/4c1bc7764c5d11f089dfae2447f2c79f.log', 'NEWSPIDER_MODULE': 'houses.spiders', 'RANDOMIZE_DOWNLOAD_DELAY': False, 'SPIDER_MODULES': ['houses.spiders'], 'TELNETCONSOLE_ENABLED': False, 'TWISTED_REACTOR': 'twisted.internet.epollreactor.EPollReactor', 'URLLENGTH_LIMIT': 5000, 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, ' 'like Gecko) Chrome/29.0.1547.65 Safari/537.36'} 2025-06-18 16:00:10 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'houses.middlewares.Fake404Middleware', 'crawler_utils.middlewares.MyUserAgentMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'rotating_proxies.middlewares.RotatingProxyMiddleware', 'rotating_proxies.middlewares.BanDetectionMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats', 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware'] 2025-06-18 16:00:10 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.start.StartSpiderMiddleware', 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2025-06-18 16:00:11 [crawler_utils.yaml_export_pipeline] INFO: YamlExportPipeline initialized with FTP host: crawler-data.boligzonen.dk, user: ftpuser 2025-06-18 16:00:11 [crawler_utils.yaml_export_pipeline] INFO: Output directories: ['upload/rental_nl'] 2025-06-18 16:00:11 [crawler_utils.yaml_export_pipeline] INFO: Output tempdir: /tmp/rental_nl 2025-06-18 16:00:11 [crawler_utils.yaml_export_pipeline] INFO: Using PostCodeValidator for project: rental_nl 2025-06-18 16:00:11 [scrapy.middleware] INFO: Enabled item pipelines: ['houses.pipelines.DuplicatesPipeline', 'crawler_utils.yaml_export_pipeline.YamlExportPipeline'] 2025-06-18 16:00:11 [scrapy.core.engine] INFO: Spider opened 2025-06-18 16:00:11 [py.warnings] WARNING: /usr/local/lib/python3.10/site-packages/scrapy/core/spidermw.py:433: ScrapyDeprecationWarning: houses.spiders.listings.ListingsSpider defines the deprecated start_requests() method. start_requests() has been deprecated in favor of a new method, start(), to support asynchronous code execution. start_requests() will stop being called in a future version of Scrapy. If you use Scrapy 2.13 or higher only, replace start_requests() with start(); note that start() is a coroutine (async def). If you need to maintain compatibility with lower Scrapy versions, when overriding start_requests() in a spider class, override start() as well; you can use super() to reuse the inherited start() implementation without copy-pasting. See the release notes of Scrapy 2.13 for details: https://docs.scrapy.org/en/2.13/news.html warn( 2025-06-18 16:00:11 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2025-06-18 16:00:11 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 0, unchecked: 7, reanimated: 0, mean backoff time: 0s) 2025-06-18 16:00:12 [scrapy.extensions.throttle] INFO: slot: uk.proxymesh.com | conc: 1 | delay: 2697 ms (-2302) | latency: 1583 ms | size: 3796 bytes 2025-06-18 16:00:12 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-18 16:00:12 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-18 16:00:12 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2025-06-18 16:00:13 [scrapy.extensions.throttle] INFO: slot: de.proxymesh.com | conc: 1 | delay: 2569 ms (-2430) | latency: 552 ms | size: 5534 bytes 2025-06-18 16:00:13 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-18 16:00:13 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-18 16:00:13 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.listings.nl/property/) 2025-06-18 16:00:13 [scrapy.extensions.throttle] INFO: slot: fr.proxymesh.com | conc: 1 | delay: 2579 ms (-2420) | latency: 636 ms | size: 5386 bytes 2025-06-18 16:00:13 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-18 16:00:13 [rotating_proxies.expire] DEBUG: Proxy is GOOD 2025-06-18 16:00:13 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.listings.nl/property/) 2025-06-18 16:00:13 [logger] INFO: Got result for (Weesperzijde,Amsterdam,Netherlands) from cache 2025-06-18 16:00:13 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.listings.nl/property/weesperzijde/> {'area': 135, 'balcony': True, 'description': 'Spacious furnished 3-bedroom apartment available with an ' 'amazing view over the Amstel river! The apartment has a total ' 'of 135sqm living space divided over the top two floors of the ' 'building..\n' 'Layout:.\n' 'Communal entrance, staircase to second floor..\n' 'Second floor:.\n' 'Large hallway with access to the living room. The living room ' 'at the front is very bright because of the large windows ' '(facing South West). From the living you can reach the ' 'balcony at the front side with an amazing view over the ' 'Amstel river..\n' 'The dining area is situated at the backside of the apartment ' 'and is conveniently situated next to the kitchen..\n' 'The modern kitchen is fully equipped with all modern, ' 'built-in appliances such as a large refrigerator, freezer, ' 'dishwasher, gas stove, large oven and separate microwave..\n' 'There is a separate (guest)toilet situated on this floor as ' 'well..\n' 'Third floor:.\n' 'Landing with access to all rooms..\n' 'The master bedroom is situated at the front side of the ' 'building and is equipped with a comfortable double bed and a ' 'large wardrobe..\n' 'There are two other spacious bedrooms situated at the ' 'backside, both equipped with pullout beds (to turn them into ' 'double beds) a desk & a wardrobe..\n' 'The luxurious bathroom is equipped with a large walk-in rain ' 'shower, large washbasin and towel radiator..\n' 'There is a second separate toilet and separate laundry space ' 'with washing machine and dryer situated on this floor..\n' 'Location:.\n' 'The apartment is situated on a beautiful, green street right ' 'next to the Amstel river. There are many shops, restaurants ' 'and supermarkets just around the corner..\n' 'From the apartment, the city center can be easily accessed by ' 'bike or public transportation. There is a metro station ' '(Wibautstraat) which takes you to Amstel Station or the ' 'Central Station of Amsterdam within a few minutes..\n' 'Overall, a comfortable, spacious apartment situated on a ' 'beautiful, quiet location nearby De Pijp and the city center ' 'of Amsterdam!.\n' 'NOT AVAILABLE TO SHARE WITH 3 FRIENDS/COLLEAGUES.', 'external_property_attributes': {'external_property_id': '980128275d216e81d60c0898419a9003', 'external_provider_id': 851, 'property_url': 'https://www.listings.nl/property/weesperzijde/'}, 'furnished': True, 'images': ['https://www.listings.nl/app/plugins/wp-kolibri/media/Yj3flL2OXZzboTihQTslTg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/dJrY9eZxq0i6BKX0CGkWAw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/laiTm9NhwICZKlpf7zu7A==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/IxjGSSdxUR3HqXLN4QG2rA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/duDx2LfnrxlKEa3p038P9A==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/EpdI6j3GPIzxbjvFu0FJQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/8oDoVdGQk5F+OM7O+veTkQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/ZuZ8GTc+6MVW3nLT53KHkg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/kjXUA21xkB9vQukNR7MDXw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/bz4ltbCCLk14sY3XjaSqg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/Uq1y2H1OLnCySWRd1tc6Q==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/ntQcak3kFRimcV1u7SIOg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/f7jZvE3iVuP5Wi59MRGXAg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/eK3hNH1WjB5gwL8tMecBvQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/7MEnf+scq3urjNZamBbq6Q==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/y4Is8o4tjf3tK2ktBV7KQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/8I9fVy0F7wapEraBOXgMQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/ccHhrm28lIrzv6dy0SA32A==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/jJcm65QBFctkt9rRjA9xNA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/Jb+Kyd5Ny9BHukhrh06AxA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/KX5sf7nn8v4oVrypqFgPog==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/XiQe0xeWe9u+W2yyKNbQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/wpABukbNwTmxvwouSVIe+Q==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/JKqT8dNzCfBIcPgtdcEwQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/b2ZwYZixYw7ZGnGKqGxtZw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/jgFA46ZAgjqOoWaL8dT+fQ==-huge.jpeg'], 'property_type': 'Apartment', 'rent': 3500, 'rooms': 3, 'street_name': 'Weesperzijde', 'zip_code_code': '1091'} 2025-06-18 16:00:13 [logger] INFO: Got result for (Rozenstraat,Amsterdam,Netherlands) from cache 2025-06-18 16:00:13 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.listings.nl/property/rozenstraat/> {'area': 85, 'description': '***NOT SUITABLE TO SHARE BETWEEN FRIENDS/COLLEAGUES DUE TO ' 'THE LAYOUT! ***.\n' 'High-quality, spacious & modern 2-bedroom apartment with an ' 'amazing common courtyard, located in the heart of the Jordaan ' 'area..\n' 'The apartment is situated in the building called, ‘De Roos’, ' 'which was built in 2007..\n' 'The property is therefore very well insulated (energy label ' 'A) and is heated by underfloor heating throughout..\n' 'Layout:.\n' 'Entrance on the ground floor, hallway with access to the ' 'living area..\n' 'The open living area is situated at the backside of the ' 'building. The living area has sliding doors that give access ' 'to the terrace and common courtyard. The courtyard is very ' 'quiet and from here you can enjoy the afternoon sun..\n' 'There is a luxurious kitchen installed on the front side. The ' 'kitchen is equipped with all modern, built-in appliances such ' 'as a (combi)oven/microwave, large refrigerator/freezer, ' 'dishwasher, 5 burner gas stove and extractor hood..\n' 'Souterrain level.\n' 'Through the staircase on the right side of the living area, ' 'you reach the souterrain level..\n' 'There is a separate toilet and small storage..\n' 'At the rear of the apartment you will find the spacious main ' 'bedroom. The master bedroom is provided with a large double ' 'bed and a walk-in closet..\n' 'Next to the main bedroom, you can find the luxurious ' 'bathroom. The bathroom is provided with a large bathtub, ' 'walk-in shower, design radiator and underfloor heating..\n' 'At the front side, you can find the second spacious bedroom. ' 'This bedroom is big enough to fit a double bed and wardrobe ' 'space..\n' 'There is a separate laundry space with the washing ' 'machine/dryer combination..\n' 'Location:.\n' 'The apartment is located on a prime location in the Jordaan ' 'area, nearby the Rozengracht, Prinsengracht, Westertoren and ' 'the Lijnbaansgracht. Many amenities such as restaurants, ' 'cafes, shops, the 9-streets area are located within walking ' 'distance..\n' 'There are many public transport options in the direct ' 'vicinity towards Central Station, Amsterdam West, ' 'Amsterdam-Zuid and Schiphol Airport.', 'energy_label': 'a_2015', 'external_property_attributes': {'external_property_id': '04079a9b193a5ca7c08b18b70ad75e95', 'external_provider_id': 851, 'property_url': 'https://www.listings.nl/property/rozenstraat/'}, 'furnished': True, 'images': ['https://www.listings.nl/app/plugins/wp-kolibri/media/ke0tTnkEVBD6OlzdvZhCg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/5V+XaWxrtD5dqGfWarGJQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/5mpusvx0u2eKECKZw2RQw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/NvrPzEthEnEu8NUtOnr9w==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/RUQPwHhthmv+JZcirF8DCA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/jv38qSYH1nYY+xD6j1dDA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/5r59Vo5EFItWRcYw5I79gA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/Magbrw0Gm9IfWtdUbl03Zg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/guA1KkLL4U7AMMQXQg5YpQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/8DhxWYB0kLy4D496iAZA2w==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/byZNkOS0zfwgtrW6ahkJ1Q==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/yzk88qI2+pEtJLUKcLhbSw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/ndeN79ENd4at3by8zeOVg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/lEyuMMxHG5SI4CfFtc3Evg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/eRKzcU2JSeGFD78vWzJxTw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/YtvosydBLPz+9TWDJBtFw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/912uNwzEGX9Boox6y8p47w==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/9zwzFVaP9VAPof6Upb20+w==-huge.jpeg'], 'property_type': 'Apartment', 'rent': 2650, 'rooms': 2, 'street_name': 'Rozenstraat', 'zip_code_code': '1016'} 2025-06-18 16:00:18 [scrapy.extensions.throttle] INFO: slot: de.proxymesh.com | conc: 1 | delay: 1347 ms (-1221) | latency: 507 ms | size: 5660 bytes 2025-06-18 16:00:18 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-18 16:00:18 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.listings.nl/property/) 2025-06-18 16:00:18 [logger] INFO: Got result for (Da Costastraat,Amsterdam,Netherlands) from cache 2025-06-18 16:00:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.listings.nl/property/da-costastraat/> {'area': 93, 'balcony': True, 'description': '***We have received a very high number of responses to this ' 'offer and will not be making any additional arrangements at ' 'the address for the time being.***.\n' 'Amazing apartment (93sqm) on the 2nd floor with a spacious ' 'balcony in the popular Old West neighborhood. The apartment ' 'will be delivered fully furnished and is great for a single ' 'expat or couple..\n' 'Layout:.\n' 'Communal entrance with large staircase to the second floor..\n' 'Entrance on the second floor:.\n' 'Large hallway with access to all rooms..\n' 'The grand living room is situated at the backside of the ' 'building, and has two doors opening on to the spacious ' 'balcony (facing West). The living room is equipped with a ' 'large 3-seat sofa, a comfortable armchair and a nice round ' 'dining table with 4 chairs..\n' 'The modern kitchen is provided with all modern, built-in ' 'appliances such as a dishwasher, combination oven/microwave, ' 'gas-stove, refrigerator and freezer..\n' 'There is an open patio in the middle that invites a lot of ' 'natural light into the apartment and creates coziness with a ' 'nice surprise effect..\n' 'The master bedroom is situated at the front side of the ' 'building and is overlooking the quiet Da Costastraat. The ' 'master bedroom is furnished with a large double bed, a ' 'wardrobe and desk. There are three large windows, all ' 'double-glazed..\n' 'The second (guest)bedroom or study is currently unfurnished ' 'and has some shelves. The bedroom.\n' 'In between both bedrooms, we find the modern bathroom..\n' 'The bathroom has a walk-in shower, toilet, washbasin and the ' 'washing machine + dryer..\n' 'There is a separate guest toilet in the hallway..\n' 'There is a private storage room in on lower floor beneath the ' 'building..\n' 'Location:.\n' 'The apartment is located in one of the most popular areas in ' 'Amsterdam at a stone’s throw away from the Jordan area/ City ' 'Centre and Kinkerstraat with its shops, daily fresh market ' 'and Foodhallen. Excellent public transport with connections ' 'to several parts of the City.', 'external_property_attributes': {'external_property_id': '6e4e7fef45df5ab929f55452b52ef924', 'external_provider_id': 851, 'property_url': 'https://www.listings.nl/property/da-costastraat/'}, 'furnished': True, 'images': ['https://www.listings.nl/app/plugins/wp-kolibri/media/XrSgD0JRX6OBfn6wJEG6pQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/d1UeuORy23BAlLzdCkKpYw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/5Riofsk3fUBFeFXMoraCMA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/yBA8JAOMuEHPljcG7jAGkg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/RtFfLmdk+ImrZRdYSTJLlg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/xBjjo86YA7sZA4eQJdctDg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/o3O+41cRmXnqCMS9u2xnew==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/m55Lq576z9+WxwbQa307g==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/6Ba2W5VmAWVQBomtL999Qw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/kMzoDOhlQAhDH70ODVNzPw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/hGkH9uN6BBCvFzAEr6BVgg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/mIIPxYitIgKizJoD7RaZbA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/XgbH72igSXFsSU5qw9Vjhw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/F5sY4O2k92mRYYFNUfwqQQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/RAgEyT7nOx4qxsvsUee4Cg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/3lFILBJICjl1q8AWp57iA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/SqlnfWicFhPk0QyUAG+ibQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/erJU76DcCU7uVbuM6MYeZw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/I7QYQgVZs7LQ3z33ZiFykQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/jWrTgRZR5JOp7s8hR4Oxw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/Pv5IKzYMuw42OFz9m9FoJQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/UtJyXlCCI+L6XzFK+VFBQQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/XqVF4bhcfP7f3NaYMaZGbA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/ACzx9yHRanYJj3MN4A5WQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/mandeGzunn84VyZ8F4PDUQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/d6K6Wml4CHDmkMuuSrkpg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/WNokyIhTHIRSZM1DOF67kQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/+vcrIE5eIHfmiZHL9HUeFA==-huge.jpeg'], 'property_type': 'Apartment', 'rent': 2600, 'rooms': 2, 'street_name': 'Da Costastraat', 'zip_code_code': '1053'} 2025-06-18 16:00:21 [scrapy.extensions.throttle] INFO: slot: de.proxymesh.com | conc: 1 | delay: 735 ms (-612) | latency: 495 ms | size: 5107 bytes 2025-06-18 16:00:21 [crawler_utils.httpcache] INFO: ***Dont cache 2025-06-18 16:00:21 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://www.listings.nl/property/) 2025-06-18 16:00:21 [logger] INFO: Got result for (Mt. Lincolnweg,Amsterdam,Netherlands) from cache 2025-06-18 16:00:21 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.listings.nl/property/mt-lincolnweg/> {'area': 72, 'description': 'WELCOME TO NAUTISCH AT NDSM!.\n' 'We have a beautiful, spacious (72sqm) 2-bedroom apartment ' 'available situated with lateral views over the IJ-river!.\n' 'The apartment will be delivered in an unfurnished condition, ' 'including floor, fully equipped kitchen, curtains and ceiling ' 'lights..\n' 'The apartment is available per July 1st for an indefinite ' 'period, minimum one year..\n' 'The private parking space in the garage is being rented ' 'together with the apartment and costs € 200,- per month..\n' 'Energy rating A++..\n' 'NO SHARING, NO STUDENTS, NO GUARANTORS..\n' 'Location:.\n' 'NDSM is increasingly becoming a modern city wharf in ' 'Amsterdam North..\n' 'New residential buildings and facilities are emerging along ' 'the IJ-river, between historic buildings of the former ' 'shipyard. The NDSM is a wonderful place to stay. Discover ' 'what NDSM has to offer you now and in the future and visit ' 'the new residential buildings, restaurants, marinas, ' 'companies and the cultural city square..\n' 'You can easily reach the city center by using the ferry, only ' '6 minutes to Pontsteiger and just 14 minutes to the Central ' 'Station of Amsterdam..\n' 'Lees minder', 'external_property_attributes': {'external_property_id': '8ea97c4ef2f0f6359a1f879548edae6e', 'external_provider_id': 851, 'property_url': 'https://www.listings.nl/property/mt-lincolnweg/'}, 'furnished': False, 'images': ['https://www.listings.nl/app/plugins/wp-kolibri/media/K2kMsB1ZtVTfyrTn8EFdQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/2vBxQe0phsaeRJYXAfqPVQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/tOuLYhiWCXuU+CLDN5sKJQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/LKMBeOJUnWoCu6XPgKAUiQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/P7qRz5LEJiqlLrEQ8dcD4w==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/sArwz6p4qXwQszxTHgcxIg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/MQAnwevzypZla3cWo0cQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/2Oe6SDLcc2dCxQ5BJRlvIA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/T1Qx6N9qZiIFjHNOG+2Qw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/licEjos5yROzP9P8BRR07g==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/uFouGmplObGsll7rF71mgA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/JO6MpRitBefLPz9wJNXjRQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/fO6A9yDhJ6zO49OU5QPXw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/ZNEy+VQFqbNU54XMc0frsQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/5KOE4YT3IyX61PtnkzuxtA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/ZqbbOrFDvzb5gjOc8HBlBw==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/XThaxSmLIws9vdfUQ8EQ==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/1vqJtzNYgyNgRNgfvCsC1Q==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/KG4fvc29bQLmdgAyyUKrA==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/7UWz65kGcophlRbtTmOyBg==-huge.jpeg', 'https://www.listings.nl/app/plugins/wp-kolibri/media/TtM+IuWOFtNu+8HI0lLBw==-huge.jpeg'], 'property_type': 'Apartment', 'rent': 2350, 'rooms': 1, 'street_name': 'Mt. Lincolnweg', 'zip_code_code': '1033'} 2025-06-18 16:00:21 [scrapy.core.engine] INFO: Closing spider (finished) 2025-06-18 16:00:21 [crawler_utils.yaml_export_pipeline] INFO: Uploading /tmp/rental_nl/listings_1750262413.yaml to /upload/rental_nl 2025-06-18 16:00:21 [crawler_utils.yaml_export_pipeline] INFO: FTP host: crawler-data.boligzonen.dk, user: ftpuser 2025-06-18 16:00:21 [paramiko.transport] DEBUG: starting thread (client mode): 0xc987dc00 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Local version/idstring: SSH-2.0-paramiko_3.5.1 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.11 2025-06-18 16:00:21 [paramiko.transport] INFO: Connected (version 2.0, client OpenSSH_9.6p1) 2025-06-18 16:00:21 [paramiko.transport] DEBUG: === Key exchange possibilities === 2025-06-18 16:00:21 [paramiko.transport] DEBUG: kex algos: sntrup761x25519-sha512@openssh.com, curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group14-sha256, ext-info-s, kex-strict-s-v00@openssh.com 2025-06-18 16:00:21 [paramiko.transport] DEBUG: server key: rsa-sha2-512, rsa-sha2-256, ecdsa-sha2-nistp256, ssh-ed25519 2025-06-18 16:00:21 [paramiko.transport] DEBUG: client encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com 2025-06-18 16:00:21 [paramiko.transport] DEBUG: server encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com 2025-06-18 16:00:21 [paramiko.transport] DEBUG: client mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1 2025-06-18 16:00:21 [paramiko.transport] DEBUG: server mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1 2025-06-18 16:00:21 [paramiko.transport] DEBUG: client compress: none, zlib@openssh.com 2025-06-18 16:00:21 [paramiko.transport] DEBUG: server compress: none, zlib@openssh.com 2025-06-18 16:00:21 [paramiko.transport] DEBUG: client lang: 2025-06-18 16:00:21 [paramiko.transport] DEBUG: server lang: 2025-06-18 16:00:21 [paramiko.transport] DEBUG: kex follows: False 2025-06-18 16:00:21 [paramiko.transport] DEBUG: === Key exchange agreements === 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Strict kex mode: True 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Kex: curve25519-sha256@libssh.org 2025-06-18 16:00:21 [paramiko.transport] DEBUG: HostKey: ssh-ed25519 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Cipher: aes128-ctr 2025-06-18 16:00:21 [paramiko.transport] DEBUG: MAC: hmac-sha2-256 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Compression: none 2025-06-18 16:00:21 [paramiko.transport] DEBUG: === End of kex handshake === 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Resetting outbound seqno after NEWKEYS due to strict mode 2025-06-18 16:00:21 [paramiko.transport] DEBUG: kex engine KexCurve25519 specified hash_algo 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Switch to new keys ... 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Resetting inbound seqno after NEWKEYS due to strict mode 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Attempting password auth... 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Got EXT_INFO: {'server-sig-algs': b'ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256', 'publickey-hostbound@openssh.com': b'0', 'ping@openssh.com': b'0'} 2025-06-18 16:00:21 [paramiko.transport] DEBUG: userauth is OK 2025-06-18 16:00:21 [paramiko.transport] INFO: Authentication (password) successful! 2025-06-18 16:00:21 [paramiko.transport] DEBUG: [chan 0] Max packet in: 32768 bytes 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Received global request "hostkeys-00@openssh.com" 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Rejecting "hostkeys-00@openssh.com" global request from server. 2025-06-18 16:00:21 [paramiko.transport] DEBUG: [chan 0] Max packet out: 32768 bytes 2025-06-18 16:00:21 [paramiko.transport] DEBUG: Secsh channel 0 opened. 2025-06-18 16:00:21 [paramiko.transport] DEBUG: [chan 0] Sesch channel 0 request ok 2025-06-18 16:00:21 [paramiko.transport.sftp] INFO: [chan 0] Opened sftp connection (server version 3) 2025-06-18 16:00:21 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_nl/listings_1750262413.yaml', 'wb') 2025-06-18 16:00:21 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_nl/listings_1750262413.yaml', 'wb') -> 00000000 2025-06-18 16:00:21 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000) 2025-06-18 16:00:21 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/rental_nl/listings_1750262413.yaml') 2025-06-18 16:00:21 [paramiko.transport.sftp] INFO: [chan 0] sftp session closed. 2025-06-18 16:00:21 [paramiko.transport] DEBUG: [chan 0] EOF sent (0) 2025-06-18 16:00:21 [crawler_utils.yaml_export_pipeline] INFO: Successfully uploaded /tmp/rental_nl/listings_1750262413.yaml to upload/rental_nl 2025-06-18 16:00:21 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 1629, 'downloader/request_count': 5, 'downloader/request_method_count/GET': 5, 'downloader/response_bytes': 26993, 'downloader/response_count': 5, 'downloader/response_status_count/200': 5, 'elapsed_time_seconds': 9.904684, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2025, 6, 18, 16, 0, 21, 139889, tzinfo=datetime.timezone.utc), 'httpcache/firsthand': 5, 'httpcache/miss': 5, 'httpcache/uncacheable': 5, 'httpcompression/response_bytes': 110879, 'httpcompression/response_count': 5, 'item_scraped_count': 4, 'items_per_minute': 26.666666666666668, 'log_count/DEBUG': 54, 'log_count/INFO': 34, 'log_count/WARNING': 1, 'memusage/max': 96559104, 'memusage/startup': 96559104, 'proxies/good': 3, 'proxies/mean_backoff': 0.0, 'proxies/reanimated': 0, 'proxies/unchecked': 4, 'request_depth_max': 1, 'response_received_count': 5, 'responses_per_minute': 33.333333333333336, 'scheduler/dequeued': 5, 'scheduler/dequeued/memory': 5, 'scheduler/enqueued': 5, 'scheduler/enqueued/memory': 5, 'start_time': datetime.datetime(2025, 6, 18, 16, 0, 11, 235205, tzinfo=datetime.timezone.utc)} 2025-06-18 16:00:21 [paramiko.transport] DEBUG: EOF in transport thread 2025-06-18 16:00:21 [scrapy.core.engine] INFO: Spider closed (finished)