Desktop version Jobs
  • Analysis
  • Categories
  • Logs
  • Crawler.stats
  • projectrental_se
    spidertvastenar
    job868bfc5c434b11f084211a70eab55c2b
    first_log_time2025-06-07 03:01:42
    latest_log_time2025-06-07 03:01:57
    runtime0:00:15
    crawled_pages 29
    scraped_items 28
    shutdown_reasonN/A
    finish_reasonfinished
    log_critical_count0
    log_error_count0
    log_warning_count0
    log_redirect_count0
    log_retry_count0
    log_ignore_count0
    latest_crawl
    latest_scrape
    latest_log
    current_time
    latest_item{'area': 44, 'available': '2025-09-01', 'external_property_attributes': {'external_property_id': 'hjeltegatan-15a-lgh-1201-i-amal', 'external_provider_id': 459, 'property_url': 'https://tvastenar.com/ledigalagenheter/hjeltegatan-15a-lgh-1201-i-amal/'}, 'heading': '', 'house_number': 15, 'images': ['https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-6-scaled.jpg.webp', 'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-4-scaled.jpg.webp', 'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-2-scaled.jpg.webp', 'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-5-scaled.jpg.webp', 'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-3-scaled.jpg.webp', 'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-7-scaled.jpg.webp', 'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-1-scaled.jpg.webp', 'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-8-scaled.jpg.webp'], 'property_type': 'Apartment', 'rent': 5169, 'rooms': 1, 'street_name': 'Hjeltegatan', 'zip_code_code': '66230'}

    • WARNING+

      INFO

      DEBUG

    • latest_crawl

      2025-06-07 03:01:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tvastenar.com/ledigalagenheter/hjeltegatan-15a-lgh-1201-i-amal/> (referer: https://tvastenar.com/ledigalagenheter/)

    • latest_item

      {'area': 44,
 'available': '2025-09-01',
 'external_property_attributes': {'external_property_id': 'hjeltegatan-15a-lgh-1201-i-amal',
                                  'external_provider_id': 459,
                                  'property_url': 'https://tvastenar.com/ledigalagenheter/hjeltegatan-15a-lgh-1201-i-amal/'},
 'heading': '',
 'house_number': 15,
 'images': ['https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-6-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-4-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-2-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-5-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-3-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-7-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-1-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-8-scaled.jpg.webp'],
 'property_type': 'Apartment',
 'rent': 5169,
 'rooms': 1,
 'street_name': 'Hjeltegatan',
 'zip_code_code': '66230'}

    • latest_scrape

      2025-06-07 03:01:56 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tvastenar.com/ledigalagenheter/hjeltegatan-15a-lgh-1201-i-amal/>

    • latest_stat

      2025-06-07 03:01:42 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

    • scrapy_version

      2.13.1

    • Head

      2025-06-07 03:01:42 [scrapy.utils.log] INFO: Scrapy 2.13.1 started (bot: scrapybot)
2025-06-07 03:01:42 [scrapy.utils.log] INFO: Versions:
{'lxml': '5.4.0',
 'libxml2': '2.13.8',
 'cssselect': '1.3.0',
 'parsel': '1.10.0',
 'w3lib': '2.3.1',
 'Twisted': '24.11.0',
 'Python': '3.10.17 (main, May 21 2025, 23:32:05) [GCC 12.2.0]',
 'pyOpenSSL': '25.1.0 (OpenSSL 3.5.0 8 Apr 2025)',
 'cryptography': '45.0.3',
 'Platform': 'Linux-6.8.0-57-generic-x86_64-with-glibc2.36'}
2025-06-07 03:01:42 [scrapy.addons] INFO: Enabled addons:
[]
2025-06-07 03:01:42 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2025-06-07 03:01:42 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.throttle.AutoThrottle',
 'scrapy.extensions.corestats.CoreStats',
 'crawler_utils.ftp_log_uploader.FTPLogUploader']
2025-06-07 03:01:42 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_DEBUG': True,
 'AUTOTHROTTLE_ENABLED': True,
 'AUTOTHROTTLE_TARGET_CONCURRENCY': 3.0,
 'COMMANDS_MODULE': 'houses.commands',
 'CONCURRENT_REQUESTS': 6,
 'CONCURRENT_REQUESTS_PER_DOMAIN': 6,
 'DEFAULT_ITEM_CLASS': 'houses.items.HousesItem',
 'FTP_PASSWORD': 'H4KgYJ4zNyTTVokQGa_4',
 'FTP_USER': 'ftpuser',
 'HTTPCACHE_DIR': '/etc/scrapyd_cache',
 'HTTPCACHE_ENABLED': True,
 'HTTPCACHE_POLICY': 'crawler_utils.httpcache.GradualExpiresPolicy',
 'HTTPCACHE_STORAGE': 'crawler_utils.httpcache.FilesystemCacheStorage',
 'LOG_FILE': '/app/logs/rental_se/tvastenar/868bfc5c434b11f084211a70eab55c2b.log',
 'NEWSPIDER_MODULE': 'houses.spiders',
 'RANDOMIZE_DOWNLOAD_DELAY': False,
 'SPIDER_MODULES': ['houses.spiders'],
 'TELNETCONSOLE_ENABLED': False,
 'TWISTED_REACTOR': 'twisted.internet.epollreactor.EPollReactor',
 'URLLENGTH_LIMIT': 5000,
 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, '
               'like Gecko) Chrome/29.0.1547.65 Safari/537.36'}
2025-06-07 03:01:42 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'houses.middlewares.ProxyMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'crawler_utils.middlewares.MyUserAgentMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats',
 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware']
2025-06-07 03:01:42 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.start.StartSpiderMiddleware',
 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2025-06-07 03:01:42 [crawler_utils.yaml_export_pipeline] INFO: YamlExportPipeline initialized with FTP host: crawler-data.boligzonen.dk, user: ftpuser
2025-06-07 03:01:42 [crawler_utils.yaml_export_pipeline] INFO: Output directories: ['upload/rental_se']
2025-06-07 03:01:42 [crawler_utils.yaml_export_pipeline] INFO: Output tempdir: /tmp/rental_se
2025-06-07 03:01:42 [crawler_utils.yaml_export_pipeline] INFO: Using PostCodeValidator for project: rental_se
2025-06-07 03:01:42 [scrapy.middleware] INFO: Enabled item pipelines:
['houses.pipelines.DuplicatesPipeline',
 'crawler_utils.yaml_export_pipeline.YamlExportPipeline']
2025-06-07 03:01:42 [scrapy.core.engine] INFO: Spider opened
2025-06-07 03:01:42 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2025-06-07 03:01:43 [scrapy.extensions.throttle] INFO: slot: tvastenar.com | conc: 1 | delay: 2565 ms (-2434) | latency:  391 ms | size: 17459 bytes
2025-06-07 03:01:43 [crawler_utils.httpcache] INFO: ***Dont cache
2025-06-07 03:01:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tvastenar.com/ledigalagenheter/> (referer: None)
2025-06-07 03:01:48 [scrapy.extensions.throttle] INFO: slot: tvastenar.com | conc: 1 | delay: 1336 ms (-1228) | latency:  324 ms | size: 24080 bytes
2025-06-07 03:01:48 [crawler_utils.httpcache] INFO: ***Dont cache
2025-06-07 03:01:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tvastenar.com/ledigalagenheter/torggatan-20-lgh-1103-i-amal-balkong/> (referer: https://tvastenar.com/ledigalagenheter/)
2025-06-07 03:01:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tvastenar.com/ledigalagenheter/torggatan-20-lgh-1103-i-amal-balkong/>
{'area': 127,
 'available': '2025-06-07',
 'external_property_attributes': {'external_property_id': 'torggatan-20-lgh-1103-i-amal-balkong',
                                  'external_provider_id': 459,
                                  'property_url': 'https://tvastenar.com/ledigalagenheter/torggatan-20-lgh-1103-i-amal-balkong/'},
 'heading': '',
 'house_number': 20,
 'images': ['https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_092815-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_092952-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_092838-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_093001-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_092946-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_092921-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_092650-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_092915-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_092745-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_092905-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_092735-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2024/05/20240924_092955-scaled.jpg.webp',

    • Tail

                  'https://tvastenar.com/wp-content/smush-webp/2025/05/Vasterlanggatan-5-lgh-1101-1-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/05/Vasterlanggatan-5-lgh-1101-7-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/05/Vasterlanggatan-5-lgh-1101-2-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/05/Vasterlanggatan-5-lgh-1101-4-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/05/Vasterlanggatan-5-lgh-1101-3-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/05/Vasterlanggatan-5-lgh-1101-6-scaled.jpg.webp'],
 'property_type': 'Apartment',
 'rent': 7944,
 'rooms': 3,
 'street_name': 'Västerlånggatan',
 'zip_code_code': '66231'}
2025-06-07 03:01:55 [scrapy.extensions.throttle] INFO: slot: tvastenar.com | conc: 1 | delay:   69 ms (+0) | latency:  206 ms | size: 23847 bytes
2025-06-07 03:01:55 [crawler_utils.httpcache] INFO: ***Dont cache
2025-06-07 03:01:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tvastenar.com/ledigalagenheter/hjeltegatan-15a-lgh-1201-i-amal/> (referer: https://tvastenar.com/ledigalagenheter/)
2025-06-07 03:01:56 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tvastenar.com/ledigalagenheter/lunnegatan-11-lgh-1201-i-amal/>
{'area': 54,
 'available': '2025-10-01',
 'external_property_attributes': {'external_property_id': 'lunnegatan-11-lgh-1201-i-amal',
                                  'external_provider_id': 459,
                                  'property_url': 'https://tvastenar.com/ledigalagenheter/lunnegatan-11-lgh-1201-i-amal/'},
 'heading': '',
 'house_number': 11,
 'images': [],
 'property_type': 'Apartment',
 'rent': 5244,
 'rooms': 2,
 'street_name': 'Lunnegatan',
 'zip_code_code': '66231'}
2025-06-07 03:01:56 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tvastenar.com/ledigalagenheter/hjeltegatan-15a-lgh-1201-i-amal/>
{'area': 44,
 'available': '2025-09-01',
 'external_property_attributes': {'external_property_id': 'hjeltegatan-15a-lgh-1201-i-amal',
                                  'external_provider_id': 459,
                                  'property_url': 'https://tvastenar.com/ledigalagenheter/hjeltegatan-15a-lgh-1201-i-amal/'},
 'heading': '',
 'house_number': 15,
 'images': ['https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-6-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-4-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-2-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-5-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-3-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-7-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-1-scaled.jpg.webp',
            'https://tvastenar.com/wp-content/smush-webp/2025/06/Hjeltegatan-15A-lgh-1201-8-scaled.jpg.webp'],
 'property_type': 'Apartment',
 'rent': 5169,
 'rooms': 1,
 'street_name': 'Hjeltegatan',
 'zip_code_code': '66230'}
2025-06-07 03:01:56 [scrapy.core.engine] INFO: Closing spider (finished)
2025-06-07 03:01:56 [paramiko.transport] DEBUG: starting thread (client mode): 0x9d16a080
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Local version/idstring: SSH-2.0-paramiko_3.5.1
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.11
2025-06-07 03:01:56 [paramiko.transport] INFO: Connected (version 2.0, client OpenSSH_9.6p1)
2025-06-07 03:01:56 [paramiko.transport] DEBUG: === Key exchange possibilities ===
2025-06-07 03:01:56 [paramiko.transport] DEBUG: kex algos: sntrup761x25519-sha512@openssh.com, curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group14-sha256, ext-info-s, kex-strict-s-v00@openssh.com
2025-06-07 03:01:56 [paramiko.transport] DEBUG: server key: rsa-sha2-512, rsa-sha2-256, ecdsa-sha2-nistp256, ssh-ed25519
2025-06-07 03:01:56 [paramiko.transport] DEBUG: client encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
2025-06-07 03:01:56 [paramiko.transport] DEBUG: server encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
2025-06-07 03:01:56 [paramiko.transport] DEBUG: client mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2025-06-07 03:01:56 [paramiko.transport] DEBUG: server mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2025-06-07 03:01:56 [paramiko.transport] DEBUG: client compress: none, zlib@openssh.com
2025-06-07 03:01:56 [paramiko.transport] DEBUG: server compress: none, zlib@openssh.com
2025-06-07 03:01:56 [paramiko.transport] DEBUG: client lang: <none>
2025-06-07 03:01:56 [paramiko.transport] DEBUG: server lang: <none>
2025-06-07 03:01:56 [paramiko.transport] DEBUG: kex follows: False
2025-06-07 03:01:56 [paramiko.transport] DEBUG: === Key exchange agreements ===
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Strict kex mode: True
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Kex: curve25519-sha256@libssh.org
2025-06-07 03:01:56 [paramiko.transport] DEBUG: HostKey: ssh-ed25519
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Cipher: aes128-ctr
2025-06-07 03:01:56 [paramiko.transport] DEBUG: MAC: hmac-sha2-256
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Compression: none
2025-06-07 03:01:56 [paramiko.transport] DEBUG: === End of kex handshake ===
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Resetting outbound seqno after NEWKEYS due to strict mode
2025-06-07 03:01:56 [paramiko.transport] DEBUG: kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Switch to new keys ...
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Resetting inbound seqno after NEWKEYS due to strict mode
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Attempting password auth...
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Got EXT_INFO: {'server-sig-algs': b'ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256', 'publickey-hostbound@openssh.com': b'0', 'ping@openssh.com': b'0'}
2025-06-07 03:01:56 [paramiko.transport] DEBUG: userauth is OK
2025-06-07 03:01:56 [paramiko.transport] INFO: Authentication (password) successful!
2025-06-07 03:01:56 [paramiko.transport] DEBUG: [chan 0] Max packet in: 32768 bytes
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Received global request "hostkeys-00@openssh.com"
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Rejecting "hostkeys-00@openssh.com" global request from server.
2025-06-07 03:01:56 [paramiko.transport] DEBUG: [chan 0] Max packet out: 32768 bytes
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Secsh channel 0 opened.
2025-06-07 03:01:56 [paramiko.transport] DEBUG: [chan 0] Sesch channel 0 request ok
2025-06-07 03:01:56 [paramiko.transport.sftp] INFO: [chan 0] Opened sftp connection (server version 3)
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_se')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_se/tvastenar')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/61a47324433211f084211a70eab55c2b.log', 'wb')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/61a47324433211f084211a70eab55c2b.log', 'wb') -> 00000000
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_se/tvastenar/61a47324433211f084211a70eab55c2b.log')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/ff97d0f2432911f084211a70eab55c2b.log', 'wb')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/ff97d0f2432911f084211a70eab55c2b.log', 'wb') -> 00000000
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_se/tvastenar/ff97d0f2432911f084211a70eab55c2b.log')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/2506c03c434311f084211a70eab55c2b.log', 'wb')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/2506c03c434311f084211a70eab55c2b.log', 'wb') -> 00000000
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_se/tvastenar/2506c03c434311f084211a70eab55c2b.log')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/c3072bae433a11f084211a70eab55c2b.log', 'wb')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/c3072bae433a11f084211a70eab55c2b.log', 'wb') -> 00000000
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_se/tvastenar/c3072bae433a11f084211a70eab55c2b.log')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/9e222636432111f084211a70eab55c2b.log', 'wb')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/9e222636432111f084211a70eab55c2b.log', 'wb') -> 00000000
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_se/tvastenar/9e222636432111f084211a70eab55c2b.log')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/868bfc5c434b11f084211a70eab55c2b.log', 'wb')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_se/tvastenar/868bfc5c434b11f084211a70eab55c2b.log', 'wb') -> 00000000
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_se/tvastenar/868bfc5c434b11f084211a70eab55c2b.log')
2025-06-07 03:01:56 [paramiko.transport.sftp] DEBUG: [chan 0] listdir(b'/upload/logs/rental_se/tvastenar')
2025-06-07 03:01:56 [paramiko.transport.sftp] INFO: [chan 0] sftp session closed.
2025-06-07 03:01:56 [paramiko.transport] DEBUG: [chan 0] EOF sent (0)
2025-06-07 03:01:56 [crawler_utils.yaml_export_pipeline] INFO: Uploading /tmp/rental_se/tvastenar_1749265308.yaml to /upload/rental_se
2025-06-07 03:01:56 [crawler_utils.yaml_export_pipeline] INFO: FTP host: crawler-data.boligzonen.dk, user: ftpuser
2025-06-07 03:01:56 [paramiko.transport] DEBUG: EOF in transport thread
2025-06-07 03:01:56 [paramiko.transport] DEBUG: starting thread (client mode): 0x9d16be20
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Local version/idstring: SSH-2.0-paramiko_3.5.1
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.11
2025-06-07 03:01:56 [paramiko.transport] INFO: Connected (version 2.0, client OpenSSH_9.6p1)
2025-06-07 03:01:56 [paramiko.transport] DEBUG: === Key exchange possibilities ===
2025-06-07 03:01:56 [paramiko.transport] DEBUG: kex algos: sntrup761x25519-sha512@openssh.com, curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group14-sha256, ext-info-s, kex-strict-s-v00@openssh.com
2025-06-07 03:01:56 [paramiko.transport] DEBUG: server key: rsa-sha2-512, rsa-sha2-256, ecdsa-sha2-nistp256, ssh-ed25519
2025-06-07 03:01:56 [paramiko.transport] DEBUG: client encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
2025-06-07 03:01:56 [paramiko.transport] DEBUG: server encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
2025-06-07 03:01:56 [paramiko.transport] DEBUG: client mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2025-06-07 03:01:56 [paramiko.transport] DEBUG: server mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2025-06-07 03:01:56 [paramiko.transport] DEBUG: client compress: none, zlib@openssh.com
2025-06-07 03:01:56 [paramiko.transport] DEBUG: server compress: none, zlib@openssh.com
2025-06-07 03:01:56 [paramiko.transport] DEBUG: client lang: <none>
2025-06-07 03:01:56 [paramiko.transport] DEBUG: server lang: <none>
2025-06-07 03:01:56 [paramiko.transport] DEBUG: kex follows: False
2025-06-07 03:01:56 [paramiko.transport] DEBUG: === Key exchange agreements ===
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Strict kex mode: True
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Kex: curve25519-sha256@libssh.org
2025-06-07 03:01:56 [paramiko.transport] DEBUG: HostKey: ssh-ed25519
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Cipher: aes128-ctr
2025-06-07 03:01:56 [paramiko.transport] DEBUG: MAC: hmac-sha2-256
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Compression: none
2025-06-07 03:01:56 [paramiko.transport] DEBUG: === End of kex handshake ===
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Resetting outbound seqno after NEWKEYS due to strict mode
2025-06-07 03:01:56 [paramiko.transport] DEBUG: kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Switch to new keys ...
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Resetting inbound seqno after NEWKEYS due to strict mode
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Got EXT_INFO: {'server-sig-algs': b'ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256', 'publickey-hostbound@openssh.com': b'0', 'ping@openssh.com': b'0'}
2025-06-07 03:01:56 [paramiko.transport] DEBUG: Attempting password auth...
2025-06-07 03:01:56 [paramiko.transport] DEBUG: userauth is OK
2025-06-07 03:01:56 [paramiko.transport] INFO: Authentication (password) successful!
2025-06-07 03:01:56 [paramiko.transport] DEBUG: [chan 0] Max packet in: 32768 bytes
2025-06-07 03:01:57 [paramiko.transport] DEBUG: Received global request "hostkeys-00@openssh.com"
2025-06-07 03:01:57 [paramiko.transport] DEBUG: Rejecting "hostkeys-00@openssh.com" global request from server.
2025-06-07 03:01:57 [paramiko.transport] DEBUG: [chan 0] Max packet out: 32768 bytes
2025-06-07 03:01:57 [paramiko.transport] DEBUG: Secsh channel 0 opened.
2025-06-07 03:01:57 [paramiko.transport] DEBUG: [chan 0] Sesch channel 0 request ok
2025-06-07 03:01:57 [paramiko.transport.sftp] INFO: [chan 0] Opened sftp connection (server version 3)
2025-06-07 03:01:57 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_se/tvastenar_1749265308.yaml', 'wb')
2025-06-07 03:01:57 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_se/tvastenar_1749265308.yaml', 'wb') -> 00000000
2025-06-07 03:01:57 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-07 03:01:57 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/rental_se/tvastenar_1749265308.yaml')
2025-06-07 03:01:57 [paramiko.transport.sftp] INFO: [chan 0] sftp session closed.
2025-06-07 03:01:57 [paramiko.transport] DEBUG: [chan 0] EOF sent (0)
2025-06-07 03:01:57 [crawler_utils.yaml_export_pipeline] INFO: Successfully uploaded /tmp/rental_se/tvastenar_1749265308.yaml to upload/rental_se
2025-06-07 03:01:57 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 10982,
 'downloader/request_count': 29,
 'downloader/request_method_count/GET': 29,
 'downloader/response_bytes': 695859,
 'downloader/response_count': 29,
 'downloader/response_status_count/200': 29,
 'elapsed_time_seconds': 13.410345,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2025, 6, 7, 3, 1, 56, 103783, tzinfo=datetime.timezone.utc),
 'httpcache/firsthand': 29,
 'httpcache/miss': 29,
 'httpcache/uncacheable': 29,
 'httpcompression/response_bytes': 4926453,
 'httpcompression/response_count': 29,
 'item_scraped_count': 28,
 'items_per_minute': None,
 'log_count/DEBUG': 166,
 'log_count/INFO': 81,
 'memusage/max': 108187648,
 'memusage/startup': 108187648,
 'request_depth_max': 1,
 'response_received_count': 29,
 'responses_per_minute': None,
 'scheduler/dequeued': 29,
 'scheduler/dequeued/memory': 29,
 'scheduler/enqueued': 29,
 'scheduler/enqueued/memory': 29,
 'start_time': datetime.datetime(2025, 6, 7, 3, 1, 42, 693438, tzinfo=datetime.timezone.utc)}
2025-06-07 03:01:57 [scrapy.core.engine] INFO: Spider closed (finished)
2025-06-07 03:01:57 [paramiko.transport] DEBUG: EOF in transport thread

    • Log

      /1/log/utf8/rental_se/tvastenar/868bfc5c434b11f084211a70eab55c2b/?job_finished=True&ui=mobile

    • Source

      http://c1.boligzonen.dk:80/logs/rental_se/tvastenar/868bfc5c434b11f084211a70eab55c2b.log

  • sourcelog
    last update time2025-06-07 03:01:57
    last update timestamp1749265317
    downloader / request bytes10982
    downloader / request count29
    downloader / request method count / GET29
    downloader / response bytes695859
    downloader / response count29
    downloader / response status count / 20029
    elapsed time seconds13.410345
    finish reasonfinished
    finish timedatetime.datetime(2025, 6, 7, 3, 1, 56, 103783, tzinfo=datetime.timezone.utc)
    httpcache / firsthand29
    httpcache / miss29
    httpcache / uncacheable29
    httpcompression / response bytes4926453
    httpcompression / response count29
    item scraped count28
    items per minuteNone
    log count / DEBUG166
    log count / INFO81
    memusage / max108187648
    memusage / startup108187648
    request depth max1
    response received count29
    responses per minuteNone
    scheduler / dequeued29
    scheduler / dequeued / memory29
    scheduler / enqueued29
    scheduler / enqueued / memory29
    start timedatetime.datetime(2025, 6, 7, 3, 1, 42, 693438, tzinfo=datetime.timezone.utc)