connection reset by peer

[i-sylar]

is there an option to suppress/skip " [Errno 104] connection reset by peer" as to avoid the script crashing?

[iAmG-r00t]

I also get such an error, when the site doen't respond when analyzing a bulk of urls. So it hits a certain url/domain and I think the url doesn't respond or hits a timeout then script then dies from there and produces the below error:

detection:INFO ` https://sitehere.com
Traceback (most recent call last):
  File "/home/bean/.local/bin/wad", line 8, in <module>
    sys.exit(main())
  File "/home/bean/.local/lib/python2.7/site-packages/wad/__main__.py", line 93, in main
    results = Detector().detect_multiple(urls, limit=options.limit, exclude=options.exclude, timeout=timeout)
  File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 84, in detect_multiple
    res = self.detect(url, limit, exclude, timeout)
  File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 48, in detect
    content = self.get_content(page, url)
  File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 94, in get_content
    content = page.read()
  File "/usr/lib/python2.7/socket.py", line 355, in read
    data = self._sock.recv(rbufsize)
  File "/usr/lib/python2.7/httplib.py", line 583, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 643, in _read_chunked
    value.append(self._safe_read(amt))
  File "/usr/lib/python2.7/httplib.py", line 698, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python2.7/socket.py", line 384, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.7/ssl.py", line 754, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 641, in read
    v = self._sslobj.read(len)
socket.error: [Errno 104] Connection reset by peer

Is there a way that the script can skip such urls when they don't respond and maybe place those urls in a file of not found.

[nikofil]

I've added some code to catch exceptions with the detection of a single URL and log them to stderr, and made a new release.

[i-sylar]

Maybe you can try fix this on the next update

Traceback (most recent call last):
File "/home/bean/.local/bin/wad", line 8, in
sys.exit(main())
File "/home/bean/.local/lib/python2.7/site-packages/wad/main.py", line 93, in main
results = Detector().detect_multiple(urls, limit=options.limit, exclude=options.exclude, timeout=timeout)
File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 87, in detect_multiple
res = self.detect(url, limit, exclude, timeout)
File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 58, in detect
findings = self.findings(url, page.info(), content)
File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 76, in findings
self.remove_exclusions(findings) # 'excludes'
File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 304, in remove_exclusions
excluded = self.excluded_by([f['app'] for f in findings])
File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 298, in excluded_by
[self.apps[app]['excludes'] for app in app_list if 'excludes' in self.apps[app]], [])))
KeyError: u'Vue'

[i-sylar]

The above seems to be an issue in the interpretation of the clue file, apps.json.

[i-sylar]

I also noticed another error,

Traceback (most recent call last):
File "/home/bean/.local/bin/wad", line 8, in
sys.exit(main())
File "/home/bean/.local/lib/python2.7/site-packages/wad/main.py", line 93, in main
results = Detector().detect_multiple(urls, limit=options.limit, exclude=options.exclude, timeout=timeout)
File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 86, in detect_multiple
res = self.detect(url, limit, exclude, timeout)
File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 57, in detect
findings = self.findings(url, page.info(), content)
File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 71, in findings
findings += self.check_html(content) # 'html'
File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 188, in check_html
return self.check_tag(data=content, key='html', key_re='html_re', show_match_only=True)
File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 181, in check_tag
show_match_only)
File "/home/bean/.local/lib/python2.7/site-packages/wad/detection.py", line 139, in check_re
show_text = ''.join(show_text.splitlines())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 21: ordinal not in range(128)

[i-sylar]

I did a work around of importing sys and setting the default encoding to utf-8. Kind of fixed this error

[nikofil]

Could you tell me how to reproduce that last one?