Desktop version Jobs
  • Analysis
  • Categories
  • Logs
  • Crawler.stats
  • projectrental_dk
    spidertasfo.dk
    job3e3ae24a48ed11f08a4d1e4a116487e0
    first_log_time2025-06-14 07:10:47
    latest_log_time2025-06-14 07:11:12
    runtime0:00:25
    crawled_pages 5
    scraped_items 4
    shutdown_reasonN/A
    finish_reasonfinished
    log_critical_count0
    log_error_count0
    log_warning_count0
    log_redirect_count3
    log_retry_count0
    log_ignore_count0
    latest_crawl
    latest_scrape
    latest_log
    current_time
    latest_item{'area': 192, 'available': None, 'balcony': True, 'deposit_numb': 47700, 'description': '.\n' 'En bolig med sjældent god placering og sjældent meget plads.\n' 'I Ryttergade, et stenkast fra Odense Banegård, udlejes en ' 'rummelig lejlighed på 192 m². Boligen ligger på 2. sal og har ' 'en stor stue med loft til kip, åben pejs og direkte adgang ' 'til en ugeneret tagterrasse..\n' 'Lejligheden er indrettet med et kontor, der har gennemgang ' 'til værelsesafdelingen. Her findes to gode værelser samt et ' 'centralt multirum, der skaber mulighed for fælles aktiviteter ' 'og samvær. Der er desuden en hems, som kan anvendes efter ' 'behov..', 'door': '', 'elevator': False, 'energy_label': 'c', 'external_property_attributes': {'external_property_id': 'a433fc4a713345eac2c592d19c7292e4', 'external_provider_id': 705, 'property_url': 'https://tasfo.dk/lejeboliger/ryttergade-14b'}, 'fixed_expanses': 1000, 'floor': 2, 'furnished': None, 'house_number': '14B', 'images': ['https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/7b85dcf8-94dc-4f35-bb83-a1a512f9da74/ESOFT_Billede_005.jpg', 'https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/31d3f6c5-4c9b-4af5-86aa-116d7342a7c2/10905732_1-328-17_photo_print.jpg', 'ENCODED:https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/1748256515167-R5RZ6RGDT4TMN51BMXJL/PLANTEGNING+P%C3%85+VEJ+%282%29.png', 'https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/1742561929747-JZADPKVWRY5UZFVNOMP4/ESOFT_Billede_004.jpg', 'https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/1742561957896-ZS4AWAFWUO148QRHRAD0/ESOFT_Billede_025.jpg', 'https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/1742561967621-9YNK5XUZCDW0F8N5AY0T/ESOFT_Billede_002.jpg'], 'pets_allowed': None, 'property_type': 'Apartment', 'rent': 15900, 'rent_in_advance_numb': 47700, 'rental_period_id': None, 'rooms': 5, 'sharing_friendly': None, 'side': '', 'street_name': 'Ryttergade', 'zip_code_code': 5000}

    • WARNING+

      INFO

    • redirect_logs
      3 in total 

      2025-06-14 07:10:48 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://tasfo.dk/lejeboliger/ryttergade-12b> from <GET https://www.tasfo.dk/lejeboliger/ryttergade-12b>
      2025-06-14 07:10:53 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://tasfo.dk/lejeboliger/munke-mose-alle-1-n-1-mf> from <GET https://www.tasfo.dk/lejeboliger/munke-mose-alle-1-n-1-mf>
      2025-06-14 07:10:58 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://tasfo.dk/lejeboliger/ryttergade-14b> from <GET https://www.tasfo.dk/lejeboliger/ryttergade-14b>

      DEBUG

    • latest_crawl

      2025-06-14 07:11:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tasfo.dk/lejeboliger/ryttergade-14b> (referer: https://tasfo.dk/lejeboliger)

    • latest_item

      {'area': 192,
 'available': None,
 'balcony': True,
 'deposit_numb': 47700,
 'description': '.\n'
                'En bolig med sjældent god placering og sjældent meget plads.\n'
                'I Ryttergade, et stenkast fra Odense Banegård, udlejes en '
                'rummelig lejlighed på 192 m². Boligen ligger på 2. sal og har '
                'en stor stue med loft til kip, åben pejs og direkte adgang '
                'til en ugeneret tagterrasse..\n'
                'Lejligheden er indrettet med et kontor, der har gennemgang '
                'til værelsesafdelingen. Her findes to gode værelser samt et '
                'centralt multirum, der skaber mulighed for fælles aktiviteter '
                'og samvær. Der er desuden en hems, som kan anvendes efter '
                'behov..',
 'door': '',
 'elevator': False,
 'energy_label': 'c',
 'external_property_attributes': {'external_property_id': 'a433fc4a713345eac2c592d19c7292e4',
                                  'external_provider_id': 705,
                                  'property_url': 'https://tasfo.dk/lejeboliger/ryttergade-14b'},
 'fixed_expanses': 1000,
 'floor': 2,
 'furnished': None,
 'house_number': '14B',
 'images': ['https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/7b85dcf8-94dc-4f35-bb83-a1a512f9da74/ESOFT_Billede_005.jpg',
            'https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/31d3f6c5-4c9b-4af5-86aa-116d7342a7c2/10905732_1-328-17_photo_print.jpg',
            'ENCODED:https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/1748256515167-R5RZ6RGDT4TMN51BMXJL/PLANTEGNING+P%C3%85+VEJ+%282%29.png',
            'https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/1742561929747-JZADPKVWRY5UZFVNOMP4/ESOFT_Billede_004.jpg',
            'https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/1742561957896-ZS4AWAFWUO148QRHRAD0/ESOFT_Billede_025.jpg',
            'https://images.squarespace-cdn.com/content/v1/636b56b114f92c1325255edc/1742561967621-9YNK5XUZCDW0F8N5AY0T/ESOFT_Billede_002.jpg'],
 'pets_allowed': None,
 'property_type': 'Apartment',
 'rent': 15900,
 'rent_in_advance_numb': 47700,
 'rental_period_id': None,
 'rooms': 5,
 'sharing_friendly': None,
 'side': '',
 'street_name': 'Ryttergade',
 'zip_code_code': 5000}

    • latest_scrape

      2025-06-14 07:11:10 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tasfo.dk/lejeboliger/ryttergade-14b>

    • latest_stat

      2025-06-14 07:10:47 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

    • scrapy_version

      2.13.2

    • Head

      2025-06-14 07:10:47 [scrapy.utils.log] INFO: Scrapy 2.13.2 started (bot: scrapybot)
2025-06-14 07:10:47 [scrapy.utils.log] INFO: Versions:
{'lxml': '5.4.0',
 'libxml2': '2.13.8',
 'cssselect': '1.3.0',
 'parsel': '1.10.0',
 'w3lib': '2.3.1',
 'Twisted': '25.5.0',
 'Python': '3.10.18 (main, Jun 10 2025, 23:52:59) [GCC 12.2.0]',
 'pyOpenSSL': '25.1.0 (OpenSSL 3.5.0 8 Apr 2025)',
 'cryptography': '45.0.4',
 'Platform': 'Linux-6.8.0-60-generic-x86_64-with-glibc2.36'}
2025-06-14 07:10:47 [crawler_utils.CommonAISpider] DEBUG: Opened new DB connection for thread 139285238692736
2025-06-14 07:10:47 [crawler_utils.CommonAISpider] INFO: Database tables checked/created.
2025-06-14 07:10:47 [scrapy.addons] INFO: Enabled addons:
[]
2025-06-14 07:10:47 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2025-06-14 07:10:47 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.throttle.AutoThrottle',
 'scrapy.extensions.corestats.CoreStats',
 'crawler_utils.ftp_log_uploader.FTPLogUploader']
2025-06-14 07:10:47 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_DEBUG': True,
 'AUTOTHROTTLE_ENABLED': True,
 'AUTOTHROTTLE_TARGET_CONCURRENCY': 3.0,
 'COMMANDS_MODULE': 'houses.commands',
 'CONCURRENT_REQUESTS': 6,
 'CONCURRENT_REQUESTS_PER_DOMAIN': 6,
 'DEFAULT_ITEM_CLASS': 'houses.items.HousesItem',
 'DOWNLOAD_DELAY': 0.3,
 'FTP_PASSWORD': 'H4KgYJ4zNyTTVokQGa_4',
 'FTP_USER': 'ftpuser',
 'HTTPCACHE_DIR': '/etc/scrapyd_cache',
 'HTTPCACHE_ENABLED': True,
 'HTTPCACHE_POLICY': 'crawler_utils.httpcache.GradualExpiresPolicy',
 'HTTPCACHE_STORAGE': 'crawler_utils.httpcache.FilesystemCacheStorage',
 'LOG_FILE': '/app/logs/rental_dk/tasfo.dk/3e3ae24a48ed11f08a4d1e4a116487e0.log',
 'NEWSPIDER_MODULE': 'houses.spiders',
 'RANDOMIZE_DOWNLOAD_DELAY': False,
 'SPIDER_MODULES': ['houses.spiders'],
 'TELNETCONSOLE_ENABLED': False,
 'TWISTED_REACTOR': 'twisted.internet.epollreactor.EPollReactor',
 'URLLENGTH_LIMIT': 5000,
 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, '
               'like Gecko) Chrome/29.0.1547.65 Safari/537.36'}
2025-06-14 07:10:47 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'crawler_utils.middlewares.CookieMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats',
 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware']
2025-06-14 07:10:47 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.start.StartSpiderMiddleware',
 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2025-06-14 07:10:47 [crawler_utils.yaml_export_pipeline] INFO: YamlExportPipeline initialized with FTP host: crawler-data.boligzonen.dk, user: ftpuser
2025-06-14 07:10:47 [crawler_utils.yaml_export_pipeline] INFO: Output directories: ['upload/rental_dk', 'upload/boligsurf_rental_dk']
2025-06-14 07:10:47 [crawler_utils.yaml_export_pipeline] INFO: Output tempdir: /tmp/rental_dk
2025-06-14 07:10:47 [crawler_utils.yaml_export_pipeline] INFO: Using PostCodeValidator for project: rental_dk
2025-06-14 07:10:47 [scrapy.middleware] INFO: Enabled item pipelines:
['houses.pipelines.DuplicatesPipeline',
 'crawler_utils.yaml_export_pipeline.YamlExportPipeline']
2025-06-14 07:10:47 [scrapy.core.engine] INFO: Spider opened
2025-06-14 07:10:47 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2025-06-14 07:10:47 [scrapy.extensions.throttle] INFO: slot: tasfo.dk | conc: 1 | delay: 2519 ms (-2480) | latency:  117 ms | size: 20786 bytes
2025-06-14 07:10:47 [crawler_utils.httpcache] INFO: ***Dont cache
2025-06-14 07:10:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tasfo.dk/lejeboliger> (referer: None)
2025-06-14 07:10:48 [scrapy.extensions.throttle] INFO: slot: www.tasfo.dk | conc: 1 | delay: 5000 ms (+0) | latency:  108 ms | size:     0 bytes
2025-06-14 07:10:48 [crawler_utils.httpcache] INFO: ***Dont cache
2025-06-14 07:10:48 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://tasfo.dk/lejeboliger/ryttergade-12b> from <GET https://www.tasfo.dk/lejeboliger/ryttergade-12b>
2025-06-14 07:10:52 [scrapy.extensions.throttle] INFO: slot: tasfo.dk | conc: 1 | delay: 1264 ms (-1255) | latency:   26 ms | size: 30244 bytes
2025-06-14 07:10:52 [crawler_utils.httpcache] INFO: ***Dont cache
2025-06-14 07:10:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tasfo.dk/lejeboliger/munke-mose-alle-1-m-3-th> (referer: https://tasfo.dk/lejeboliger)
2025-06-14 07:10:53 [crawler_utils.CommonAISpider] DEBUG: Found item attributes in cache
2025-06-14 07:10:53 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tasfo.dk/lejeboliger/munke-mose-alle-1-m-3-th>
{'area': 143,
 'available': '2025-08-01',
 'balcony': True,
 'deposit_numb': 50700,
 'description': '.\n'
                'Rummelig 4-værelses på 3. sal med altan.\n'
                'Smukke rammer, høj kvalitet og god service – tæt på både '
                'byliv og natur..\n'
                'Lejligheden er indrettet i de smukke historiske '
                'skolebygninger ved Munke Mose, som i 2019 blev omdannet til '
                'boliger med respekt for det oprindelige udtryk og med fokus '
                'på høj byggeteknisk og æstetisk kvalitet..\n'

    • Tail

      2025-06-14 07:11:10 [paramiko.transport] DEBUG: starting thread (client mode): 0xd6080760
2025-06-14 07:11:10 [paramiko.transport] DEBUG: Local version/idstring: SSH-2.0-paramiko_3.5.1
2025-06-14 07:11:10 [paramiko.transport] DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.11
2025-06-14 07:11:10 [paramiko.transport] INFO: Connected (version 2.0, client OpenSSH_9.6p1)
2025-06-14 07:11:10 [paramiko.transport] DEBUG: === Key exchange possibilities ===
2025-06-14 07:11:10 [paramiko.transport] DEBUG: kex algos: sntrup761x25519-sha512@openssh.com, curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group14-sha256, ext-info-s, kex-strict-s-v00@openssh.com
2025-06-14 07:11:10 [paramiko.transport] DEBUG: server key: rsa-sha2-512, rsa-sha2-256, ecdsa-sha2-nistp256, ssh-ed25519
2025-06-14 07:11:10 [paramiko.transport] DEBUG: client encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
2025-06-14 07:11:10 [paramiko.transport] DEBUG: server encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
2025-06-14 07:11:10 [paramiko.transport] DEBUG: client mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2025-06-14 07:11:10 [paramiko.transport] DEBUG: server mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2025-06-14 07:11:10 [paramiko.transport] DEBUG: client compress: none, zlib@openssh.com
2025-06-14 07:11:10 [paramiko.transport] DEBUG: server compress: none, zlib@openssh.com
2025-06-14 07:11:10 [paramiko.transport] DEBUG: client lang: <none>
2025-06-14 07:11:10 [paramiko.transport] DEBUG: server lang: <none>
2025-06-14 07:11:10 [paramiko.transport] DEBUG: kex follows: False
2025-06-14 07:11:10 [paramiko.transport] DEBUG: === Key exchange agreements ===
2025-06-14 07:11:10 [paramiko.transport] DEBUG: Strict kex mode: True
2025-06-14 07:11:10 [paramiko.transport] DEBUG: Kex: curve25519-sha256@libssh.org
2025-06-14 07:11:10 [paramiko.transport] DEBUG: HostKey: ssh-ed25519
2025-06-14 07:11:10 [paramiko.transport] DEBUG: Cipher: aes128-ctr
2025-06-14 07:11:10 [paramiko.transport] DEBUG: MAC: hmac-sha2-256
2025-06-14 07:11:10 [paramiko.transport] DEBUG: Compression: none
2025-06-14 07:11:10 [paramiko.transport] DEBUG: === End of kex handshake ===
2025-06-14 07:11:10 [paramiko.transport] DEBUG: Resetting outbound seqno after NEWKEYS due to strict mode
2025-06-14 07:11:10 [paramiko.transport] DEBUG: kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
2025-06-14 07:11:10 [paramiko.transport] DEBUG: Switch to new keys ...
2025-06-14 07:11:10 [paramiko.transport] DEBUG: Resetting inbound seqno after NEWKEYS due to strict mode
2025-06-14 07:11:10 [paramiko.transport] DEBUG: Got EXT_INFO: {'server-sig-algs': b'ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256', 'publickey-hostbound@openssh.com': b'0', 'ping@openssh.com': b'0'}
2025-06-14 07:11:10 [paramiko.transport] DEBUG: Attempting password auth...
2025-06-14 07:11:10 [paramiko.transport] DEBUG: userauth is OK
2025-06-14 07:11:10 [paramiko.transport] INFO: Authentication (password) successful!
2025-06-14 07:11:10 [paramiko.transport] DEBUG: [chan 0] Max packet in: 32768 bytes
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Received global request "hostkeys-00@openssh.com"
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Rejecting "hostkeys-00@openssh.com" global request from server.
2025-06-14 07:11:11 [paramiko.transport] DEBUG: [chan 0] Max packet out: 32768 bytes
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Secsh channel 0 opened.
2025-06-14 07:11:11 [paramiko.transport] DEBUG: [chan 0] Sesch channel 0 request ok
2025-06-14 07:11:11 [paramiko.transport.sftp] INFO: [chan 0] Opened sftp connection (server version 3)
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_dk')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_dk/tasfo.dk')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/dc4bb68e48e411f08a4d1e4a116487e0.log', 'wb')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/dc4bb68e48e411f08a4d1e4a116487e0.log', 'wb') -> 00000000
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_dk/tasfo.dk/dc4bb68e48e411f08a4d1e4a116487e0.log')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/7bd3506c48dc11f08a4d1e4a116487e0.log', 'wb')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/7bd3506c48dc11f08a4d1e4a116487e0.log', 'wb') -> 00000000
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_dk/tasfo.dk/7bd3506c48dc11f08a4d1e4a116487e0.log')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/553bee6e48c311f0820242e93651e872.log', 'wb')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/553bee6e48c311f0820242e93651e872.log', 'wb') -> 00000000
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_dk/tasfo.dk/553bee6e48c311f0820242e93651e872.log')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/b71d1fc448cb11f0820242e93651e872.log', 'wb')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/b71d1fc448cb11f0820242e93651e872.log', 'wb') -> 00000000
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_dk/tasfo.dk/b71d1fc448cb11f0820242e93651e872.log')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/3e3ae24a48ed11f08a4d1e4a116487e0.log', 'wb')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/3e3ae24a48ed11f08a4d1e4a116487e0.log', 'wb') -> 00000000
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_dk/tasfo.dk/3e3ae24a48ed11f08a4d1e4a116487e0.log')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/195a381848d411f0b04caa5b0d4757b6.log', 'wb')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/logs/rental_dk/tasfo.dk/195a381848d411f0b04caa5b0d4757b6.log', 'wb') -> 00000000
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/logs/rental_dk/tasfo.dk/195a381848d411f0b04caa5b0d4757b6.log')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] listdir(b'/upload/logs/rental_dk/tasfo.dk')
2025-06-14 07:11:11 [paramiko.transport.sftp] INFO: [chan 0] sftp session closed.
2025-06-14 07:11:11 [paramiko.transport] DEBUG: [chan 0] EOF sent (0)
2025-06-14 07:11:11 [crawler_utils.yaml_export_pipeline] INFO: Uploading /tmp/rental_dk/tasfo.dk_1749885053.yaml to /upload/rental_dk
2025-06-14 07:11:11 [crawler_utils.yaml_export_pipeline] INFO: FTP host: crawler-data.boligzonen.dk, user: ftpuser
2025-06-14 07:11:11 [paramiko.transport] DEBUG: EOF in transport thread
2025-06-14 07:11:11 [paramiko.transport] DEBUG: starting thread (client mode): 0xd60839d0
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Local version/idstring: SSH-2.0-paramiko_3.5.1
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.11
2025-06-14 07:11:11 [paramiko.transport] INFO: Connected (version 2.0, client OpenSSH_9.6p1)
2025-06-14 07:11:11 [paramiko.transport] DEBUG: === Key exchange possibilities ===
2025-06-14 07:11:11 [paramiko.transport] DEBUG: kex algos: sntrup761x25519-sha512@openssh.com, curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group14-sha256, ext-info-s, kex-strict-s-v00@openssh.com
2025-06-14 07:11:11 [paramiko.transport] DEBUG: server key: rsa-sha2-512, rsa-sha2-256, ecdsa-sha2-nistp256, ssh-ed25519
2025-06-14 07:11:11 [paramiko.transport] DEBUG: client encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
2025-06-14 07:11:11 [paramiko.transport] DEBUG: server encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
2025-06-14 07:11:11 [paramiko.transport] DEBUG: client mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2025-06-14 07:11:11 [paramiko.transport] DEBUG: server mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2025-06-14 07:11:11 [paramiko.transport] DEBUG: client compress: none, zlib@openssh.com
2025-06-14 07:11:11 [paramiko.transport] DEBUG: server compress: none, zlib@openssh.com
2025-06-14 07:11:11 [paramiko.transport] DEBUG: client lang: <none>
2025-06-14 07:11:11 [paramiko.transport] DEBUG: server lang: <none>
2025-06-14 07:11:11 [paramiko.transport] DEBUG: kex follows: False
2025-06-14 07:11:11 [paramiko.transport] DEBUG: === Key exchange agreements ===
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Strict kex mode: True
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Kex: curve25519-sha256@libssh.org
2025-06-14 07:11:11 [paramiko.transport] DEBUG: HostKey: ssh-ed25519
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Cipher: aes128-ctr
2025-06-14 07:11:11 [paramiko.transport] DEBUG: MAC: hmac-sha2-256
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Compression: none
2025-06-14 07:11:11 [paramiko.transport] DEBUG: === End of kex handshake ===
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Resetting outbound seqno after NEWKEYS due to strict mode
2025-06-14 07:11:11 [paramiko.transport] DEBUG: kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Switch to new keys ...
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Resetting inbound seqno after NEWKEYS due to strict mode
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Attempting password auth...
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Got EXT_INFO: {'server-sig-algs': b'ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256', 'publickey-hostbound@openssh.com': b'0', 'ping@openssh.com': b'0'}
2025-06-14 07:11:11 [paramiko.transport] DEBUG: userauth is OK
2025-06-14 07:11:11 [paramiko.transport] INFO: Authentication (password) successful!
2025-06-14 07:11:11 [paramiko.transport] DEBUG: [chan 0] Max packet in: 32768 bytes
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Received global request "hostkeys-00@openssh.com"
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Rejecting "hostkeys-00@openssh.com" global request from server.
2025-06-14 07:11:11 [paramiko.transport] DEBUG: [chan 0] Max packet out: 32768 bytes
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Secsh channel 0 opened.
2025-06-14 07:11:11 [paramiko.transport] DEBUG: [chan 0] Sesch channel 0 request ok
2025-06-14 07:11:11 [paramiko.transport.sftp] INFO: [chan 0] Opened sftp connection (server version 3)
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_dk/tasfo.dk_1749885053.yaml', 'wb')
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/rental_dk/tasfo.dk_1749885053.yaml', 'wb') -> 00000000
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-14 07:11:11 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/rental_dk/tasfo.dk_1749885053.yaml')
2025-06-14 07:11:11 [paramiko.transport.sftp] INFO: [chan 0] sftp session closed.
2025-06-14 07:11:11 [paramiko.transport] DEBUG: [chan 0] EOF sent (0)
2025-06-14 07:11:11 [crawler_utils.yaml_export_pipeline] INFO: Successfully uploaded /tmp/rental_dk/tasfo.dk_1749885053.yaml to upload/rental_dk
2025-06-14 07:11:11 [crawler_utils.yaml_export_pipeline] INFO: Uploading /tmp/rental_dk/tasfo.dk_1749885053.yaml to /upload/boligsurf_rental_dk
2025-06-14 07:11:11 [crawler_utils.yaml_export_pipeline] INFO: FTP host: crawler-data.boligzonen.dk, user: ftpuser
2025-06-14 07:11:11 [paramiko.transport] DEBUG: EOF in transport thread
2025-06-14 07:11:11 [paramiko.transport] DEBUG: starting thread (client mode): 0xd60831f0
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Local version/idstring: SSH-2.0-paramiko_3.5.1
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.11
2025-06-14 07:11:11 [paramiko.transport] INFO: Connected (version 2.0, client OpenSSH_9.6p1)
2025-06-14 07:11:11 [paramiko.transport] DEBUG: === Key exchange possibilities ===
2025-06-14 07:11:11 [paramiko.transport] DEBUG: kex algos: sntrup761x25519-sha512@openssh.com, curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group14-sha256, ext-info-s, kex-strict-s-v00@openssh.com
2025-06-14 07:11:11 [paramiko.transport] DEBUG: server key: rsa-sha2-512, rsa-sha2-256, ecdsa-sha2-nistp256, ssh-ed25519
2025-06-14 07:11:11 [paramiko.transport] DEBUG: client encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
2025-06-14 07:11:11 [paramiko.transport] DEBUG: server encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
2025-06-14 07:11:11 [paramiko.transport] DEBUG: client mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2025-06-14 07:11:11 [paramiko.transport] DEBUG: server mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2025-06-14 07:11:11 [paramiko.transport] DEBUG: client compress: none, zlib@openssh.com
2025-06-14 07:11:11 [paramiko.transport] DEBUG: server compress: none, zlib@openssh.com
2025-06-14 07:11:11 [paramiko.transport] DEBUG: client lang: <none>
2025-06-14 07:11:11 [paramiko.transport] DEBUG: server lang: <none>
2025-06-14 07:11:11 [paramiko.transport] DEBUG: kex follows: False
2025-06-14 07:11:11 [paramiko.transport] DEBUG: === Key exchange agreements ===
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Strict kex mode: True
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Kex: curve25519-sha256@libssh.org
2025-06-14 07:11:11 [paramiko.transport] DEBUG: HostKey: ssh-ed25519
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Cipher: aes128-ctr
2025-06-14 07:11:11 [paramiko.transport] DEBUG: MAC: hmac-sha2-256
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Compression: none
2025-06-14 07:11:11 [paramiko.transport] DEBUG: === End of kex handshake ===
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Resetting outbound seqno after NEWKEYS due to strict mode
2025-06-14 07:11:11 [paramiko.transport] DEBUG: kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Switch to new keys ...
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Resetting inbound seqno after NEWKEYS due to strict mode
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Got EXT_INFO: {'server-sig-algs': b'ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256', 'publickey-hostbound@openssh.com': b'0', 'ping@openssh.com': b'0'}
2025-06-14 07:11:11 [paramiko.transport] DEBUG: Attempting password auth...
2025-06-14 07:11:11 [paramiko.transport] DEBUG: userauth is OK
2025-06-14 07:11:11 [paramiko.transport] INFO: Authentication (password) successful!
2025-06-14 07:11:11 [paramiko.transport] DEBUG: [chan 0] Max packet in: 32768 bytes
2025-06-14 07:11:12 [paramiko.transport] DEBUG: Received global request "hostkeys-00@openssh.com"
2025-06-14 07:11:12 [paramiko.transport] DEBUG: Rejecting "hostkeys-00@openssh.com" global request from server.
2025-06-14 07:11:12 [paramiko.transport] DEBUG: [chan 0] Max packet out: 32768 bytes
2025-06-14 07:11:12 [paramiko.transport] DEBUG: Secsh channel 0 opened.
2025-06-14 07:11:12 [paramiko.transport] DEBUG: [chan 0] Sesch channel 0 request ok
2025-06-14 07:11:12 [paramiko.transport.sftp] INFO: [chan 0] Opened sftp connection (server version 3)
2025-06-14 07:11:12 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/boligsurf_rental_dk/tasfo.dk_1749885053.yaml', 'wb')
2025-06-14 07:11:12 [paramiko.transport.sftp] DEBUG: [chan 0] open(b'/upload/boligsurf_rental_dk/tasfo.dk_1749885053.yaml', 'wb') -> 00000000
2025-06-14 07:11:12 [paramiko.transport.sftp] DEBUG: [chan 0] close(00000000)
2025-06-14 07:11:12 [paramiko.transport.sftp] DEBUG: [chan 0] stat(b'/upload/boligsurf_rental_dk/tasfo.dk_1749885053.yaml')
2025-06-14 07:11:12 [paramiko.transport.sftp] INFO: [chan 0] sftp session closed.
2025-06-14 07:11:12 [paramiko.transport] DEBUG: [chan 0] EOF sent (0)
2025-06-14 07:11:12 [crawler_utils.yaml_export_pipeline] INFO: Successfully uploaded /tmp/rental_dk/tasfo.dk_1749885053.yaml to upload/boligsurf_rental_dk
2025-06-14 07:11:12 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 3972,
 'downloader/request_count': 8,
 'downloader/request_method_count/GET': 8,
 'downloader/response_bytes': 146089,
 'downloader/response_count': 8,
 'downloader/response_status_count/200': 5,
 'downloader/response_status_count/301': 3,
 'elapsed_time_seconds': 23.007392,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2025, 6, 14, 7, 11, 10, 791107, tzinfo=datetime.timezone.utc),
 'httpcache/firsthand': 8,
 'httpcache/miss': 8,
 'httpcache/uncacheable': 8,
 'httpcompression/response_bytes': 757215,
 'httpcompression/response_count': 5,
 'item_scraped_count': 4,
 'items_per_minute': None,
 'log_count/DEBUG': 168,
 'log_count/INFO': 47,
 'memusage/max': 146968576,
 'memusage/startup': 146968576,
 'request_depth_max': 1,
 'response_received_count': 5,
 'responses_per_minute': None,
 'scheduler/dequeued': 8,
 'scheduler/dequeued/memory': 8,
 'scheduler/enqueued': 8,
 'scheduler/enqueued/memory': 8,
 'start_time': datetime.datetime(2025, 6, 14, 7, 10, 47, 783715, tzinfo=datetime.timezone.utc)}
2025-06-14 07:11:12 [scrapy.core.engine] INFO: Spider closed (finished)
2025-06-14 07:11:12 [paramiko.transport] DEBUG: EOF in transport thread

    • Log

      /2/log/utf8/rental_dk/tasfo.dk/3e3ae24a48ed11f08a4d1e4a116487e0/?job_finished=True&ui=mobile

    • Source

      http://c2.boligzonen.dk:80/logs/rental_dk/tasfo.dk/3e3ae24a48ed11f08a4d1e4a116487e0.log

  • sourcelog
    last update time2025-06-14 07:11:12
    last update timestamp1749885072
    downloader / request bytes3972
    downloader / request count8
    downloader / request method count / GET8
    downloader / response bytes146089
    downloader / response count8
    downloader / response status count / 2005
    downloader / response status count / 3013
    elapsed time seconds23.007392
    finish reasonfinished
    finish timedatetime.datetime(2025, 6, 14, 7, 11, 10, 791107, tzinfo=datetime.timezone.utc)
    httpcache / firsthand8
    httpcache / miss8
    httpcache / uncacheable8
    httpcompression / response bytes757215
    httpcompression / response count5
    item scraped count4
    items per minuteNone
    log count / DEBUG168
    log count / INFO47
    memusage / max146968576
    memusage / startup146968576
    request depth max1
    response received count5
    responses per minuteNone
    scheduler / dequeued8
    scheduler / dequeued / memory8
    scheduler / enqueued8
    scheduler / enqueued / memory8
    start timedatetime.datetime(2025, 6, 14, 7, 10, 47, 783715, tzinfo=datetime.timezone.utc)