[Bug] Several error messages when using SOCKS5 protocol by torsocks

[D3vil0p3r]

Feature Request or Bug or Other
Bug

Describe the feature request or bug or other
I know the developers are working for SOCKS protocol support on the tool. I tried to use torsocks to proxy the tool via SOCKS protocol. When I execute the tool, for some sources I get the following messages:

[*] Searching Threatminer. 
Google is blocking your ip and the workaround, returning
Searching 100 results.
An exception has occurred: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://api.sublist3r.com/search.php?domain=google.com')
[*] Searching Sublist3r. 
[*] Searching Baidu. 
Google is blocking your ip and the workaround, returning
Searching 200 results.
Google is blocking your ip and the workaround, returning
Google is blocking your ip and the workaround, returning
Searching 300 results.
Searching 100 results.
Google is blocking your ip and the workaround, returning
Google is blocking your ip and the workaround, returning
Searching 200 results.
Searching 400 results.
[*] Searching Hackertarget. 
Searching 0 results.
[*] Searching Bing. 
Searching results.
Google is blocking your ip and the workaround, returning
Google is blocking your ip and the workaround, returning
[*] Searching Virustotal. 
Searching 300 results.
Searching 500 results.
[*] Searching Linkedin. 
[*] Searching Duckduckgo. 
Google is blocking your ip and the workaround, returning
Google is blocking your ip and the workaround, returning
Searching 0 results.
Searching 400 results.
An exception has occurred: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://dns.bufferover.run/dns?q=google.com')
Google is blocking your ip and the workaround, returning
Searching 100 results.
Google is blocking your ip and the workaround, returning
Searching results.
[*] Searching Certspotter. 
Searching 500 results.
[*] Searching Linkedin. 
An exception has occurred: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://jldc.me/anubis/subdomains/google.com')
[*] Searching Anubis. 
Google is blocking your ip and the workaround, returning
Searching 200 results.
An exception has occurred: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://www.threatcrowd.org/searchApi/v2/domain/report/?domain=google.com')
string indices must be integers
[*] Searching Threatcrowd. 
[*] Searching N45ht. 
Google is blocking your ip and the workaround, returning
Searching 300 results.
[*] Searching Qwant. 
An exception has occurred: Cannot connect to host www.google.com:80 ssl:<ssl.SSLContext object at 0x75e5f9c93e40> [Temporary failure in name resolution]
An exception occurred: Cannot connect to host dnsdumpster.com:443 ssl:default [Temporary failure in name resolution]
[*] Searching Dnsdumpster. 
Searching 400 results.
An exception has occurred: 'NoneType' object has no attribute 'find'
[*] Searching Rapiddns. 
An exception has occurred: Cannot connect to host www.google.com:80 ssl:<ssl.SSLContext object at 0x75e5f9b08c40> [Temporary failure in name resolution]
Searching 500 results.
[*] Searching Google. 
An exception has occurred: Cannot connect to host crt.sh:443 ssl:<ssl.SSLContext object at 0x75e5f35036c0> [Temporary failure in name resolution]
[*] Searching CRTsh. 
An exception has occurred: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://sonar.omnisint.io/all/google.com?page=1')
[*] Searching Omnisint. 
An exception has occurred: Cannot connect to host otx.alienvault.com:443 ssl:<ssl.SSLContext object at 0x75e5f3503ac0> [Temporary failure in name resolution]
[*] Searching Urlscan. 
Google is blocking your ip and the workaround, returning
Searching 0 results.
[*] Searching Trello. 
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x75e5f9b76520>

So, regardless Google is blocking your ip... because of TOR connection, the issue messages I refer among the above ones are:

An exception has occurred: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://api.sublist3r.com/search.php?domain=google.com') (i.e. sublist3r in this case)

string indices must be integers

An exception has occurred: Cannot connect to host www.google.com:80 ssl:<ssl.SSLContext object at 0x75e5f9c93e40> [Temporary failure in name resolution]

An exception occurred: Cannot connect to host dnsdumpster.com:443 ssl:default [Temporary failure in name resolution]

An exception has occurred: 'NoneType' object has no attribute 'find'

Unclosed client session

At the end of the tool I receive some results, but I think that, because of these messages above, some sources don't contribute to the results.

To Reproduce
Steps to reproduce the behaviour:

1. Run tool like this: torsocks theHarvester -d google.com -b all
2. See error

Expected behaviour
It is expected these messages are not produced.

Screenshots
If possible please add screenshots to help explain your problem.

System Information (System that tool is running on):

• OS: Debian Bullseye

[L1ghtn1ng]

@NotoriousRebel Can you chime in on this one please?

[NotoriousRebel]

@D3vil0per Apologies to getting around to this so late, the answer is simple. Socks5 support is not currently supported; however, using a library such as aiohttp-socks and then simply modifying the async with aiohttp.ClientSession lines within the fetch_all function in core.py it could be implemented quite trivially. If verified to work, a PR can be made or we can officially add support considering it would be trivial, if the aiohttp-socks or other library on GH were to be proven to work and are stable.

[D3vil0p3r]

@D3vil0per Apologies to getting around to this so late, the answer is simple. Socks5 support is not currently supported; however, using a library such as aiohttp-socks and then simply modifying the async with aiohttp.ClientSession lines within the fetch_all function in core.py it could be implemented quite trivially. If verified to work, a PR can be made or we can officially add support considering it would be trivial, if the aiohttp-socks or other library on GH were to be proven to work and are stable.

Thank you for the answer @NotoriousRebel . Where I can find core.py? I cannot find it on the repo

[L1ghtn1ng]

It’s here theHarvester/lib/core.py Thanks, Jay On 7 Aug 2022, at 05:22, D3vil0per ***@***.***> wrote:  @D3vil0per<https://github.com/D3vil0per> Apologies to getting around to this so late, the answer is simple. Socks5 support is not currently supported; however, using a library such as aiohttp-socks<https://github.com/romis2012/aiohttp-socks> and then simply modifying the async with aiohttp.ClientSession lines within the fetch_all<https://github.com/laramies/theHarvester/blob/master/theHarvester/lib/core.py#L324> function in core.py it could be implemented quite trivially. If verified to work, a PR can be made or we can officially add support considering it would be trivial, if the aiohttp-socks or other library on GH were to be proven to work and are stable. Thank you for the answer @NotoriousRebel<https://github.com/NotoriousRebel> . Where I can find core.py? I cannot find it on the repo — Reply to this email directly, view it on GitHub<#940 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA3V2QQZ3RHPXF5L3PKZ653VX42XHANCNFSM5JFWFSAQ>. You are receiving this because you commented.Message ID: ***@***.***>