Stabilized v1.1.3

README.md

@@ -9,26 +9,43 @@
 </p>
 
 <p align="center">
-<img alt="Static Badge" src="https://img.shields.io/badge/v1.1.1-CURRENT%20STABLE%20VERSION?style=for-the-badge&label=CURRENT%20STABLE%20VERSION&color=red"> <img alt="Static Badge" src="https://img.shields.io/badge/V1.1.2-red?style=for-the-badge&logo=f&label=CURRENT%20ROLLING%20VERSION">
+<img alt="Static Badge" src="https://img.shields.io/badge/v1.1.3-CURRENT%20STABLE%20VERSION?style=for-the-badge&label=CURRENT%20STABLE%20VERSION&color=red"> <img alt="Static Badge" src="https://img.shields.io/badge/V1.1.4-red?style=for-the-badge&logo=f&label=CURRENT%20ROLLING%20VERSION">
+<img alt="Static Badge" src="https://img.shields.io/badge/DOMAIN_OSINT-CATEGORY?style=for-the-badge&label=TOOL%20CATEGORY&color=red"> <img alt="Static Badge" src="https://img.shields.io/badge/CLI-CATEGORY?style=for-the-badge&label=interface%20type&color=red">
 </p>
 
 <p align="center">
-<img alt="Static Badge" src="https://img.shields.io/badge/DOMAIN_OSINT-CATEGORY?style=for-the-badge&label=TOOL%20CATEGORY&color=red"> <img alt="Static Badge" src="https://img.shields.io/badge/CLI-CATEGORY?style=for-the-badge&label=interface%20type&color=red">
+<img alt="Static Badge" src="https://img.shields.io/badge/supports-virustotal_api-red?style=for-the-badge"> <img alt="Static Badge" src="https://img.shields.io/badge/supports-securitytrails_api-red?style=for-the-badge">
+
 </p>
 
+> Attention! DPULSE is a research tool. It is not intended for criminal activities! Use DPULSE only on allowed domains and for legal purposes!
 
-> DPULSE was created as a research tool, and it is not intended for criminal activities. Use DPULSE only on allowed domains and for legal purposes!
+# Repository map
 
-> You can visit [DPULSE wiki](https://github.com/OSINT-TECHNOLOGIES/dpulse/wiki/DPULSE-WIKI) in order to get more technical information about this project
+## What to visit?
 
-> You can visit [DPULSE roadmap](https://github.com/users/OSINT-TECHNOLOGIES/projects/1) to get more information about development process
+| What do you want to see? | Link |
+| --- | --- |
+| What is DPULSE? | [See "About DPULSE" page](https://github.com/OSINT-TECHNOLOGIES/dpulse?tab=readme-ov-file#about-dpulse) |
+| Where I can find some demo and use-cases? | [See "Demo and use-cases" page](https://github.com/OSINT-TECHNOLOGIES/dpulse?tab=readme-ov-file#dpulse-demo-and-use-cases) |
+| I want to read project documentation | [See DPULSE wiki](https://github.com/OSINT-TECHNOLOGIES/dpulse/wiki/DPULSE-WIKI) |
+| I want to see project roadmap and future development plans | [See DPULSE roadmap](https://github.com/users/OSINT-TECHNOLOGIES/projects/1) |
 
-> You can also contact the developer via e-mail: [email protected]
+## What to download?
 
-***[Download DPULSE stable ZIP archive (with latest stable changes)](https://github.com/OSINT-TECHNOLOGIES/dpulse/archive/refs/heads/main.zip)***
+| Your expectations | Version and link for you |
+| --- | --- |
+| I want to use only tested and stable version of DPULSE | [DPULSE stable ZIP archive](https://github.com/OSINT-TECHNOLOGIES/dpulse/archive/refs/heads/main.zip) |
+| I don't mind to use DPULSE with latest changes and you're OK with bugs and issues  | [DPULSE rolling ZIP archive](https://github.com/OSINT-TECHNOLOGIES/dpulse/archive/refs/heads/rolling.zip) |
+| I want to use only one specific version of DPULSE  | [See DPULSE releases page](https://github.com/OSINT-TECHNOLOGIES/dpulse/releases) |
+| I want to see more detailed installation instructions | [See DPULSE installation guides](https://github.com/OSINT-TECHNOLOGIES/dpulse?tab=readme-ov-file#how-to-install-and-run-dpulse)
 
-***[Download DPULSE rolling ZIP archive (with latest developer commit)](https://github.com/OSINT-TECHNOLOGIES/dpulse/archive/refs/heads/rolling.zip)***
+## How can I contact the developer? 
 
+| Reasons to contact | Links & addresses |
+| --- | --- |
+| I want to talk with developer in person  | DM to [email protected] |
+| I want to report some bug or issue, or maybe I have some good idea for developer  | [Make a new issue page](https://github.com/OSINT-TECHNOLOGIES/dpulse/issues/new/choose) |
 
 # About DPULSE
 
@@ -43,8 +60,8 @@ DPULSE is a software solution for conducting OSINT research in relation to a cer
    - SSL certificate info
    - possible vulnerabilities
    - open ports
-   - CPEs, used web-technologies and so on.
-   - It also can download sitemap.xml and robots.txt files from a domain, and, moreover, it can do automated Google Dorking
+   - CPEs, used web-technologies and so on
+   - It also can download sitemap.xml and robots.txt files from a domain
 
 2. ***PageSearch standard scan:*** extended subdomains deep search function, which starts in addition to basic scan and which can find:
    - more e-mail addresses
@@ -57,7 +74,10 @@ DPULSE is a software solution for conducting OSINT research in relation to a cer
 
 3. ***PageSearch Sitemap inspection scan:*** sitemap links crawler which starts in addition to basic scan and which can find even more e-mails
 
-
+4. ***Dorking scan:*** extended domain research function with prepared Google Dorking databases for different purposes, such as IoT dorking, files dorking, admin panels dorking and so on. Moreover, this mode allows you to create your own custom Google Dorking database
+
+5. ***API scan:*** extended domain research function with prepared functions for 3rd party APIs usage. Currently DPULSE supports VirusTotal API (for brief domain information gathering) and SecurityTrails API (deep subdomains and DNS enumeration)
+
 Finally, DPULSE compiles all found data into an easy-to-read PDF, HTML or XLSX report by category. It also saves all information about scan in local report storage database, which can be restored later.
 
 # How to install and run DPULSE
@@ -126,7 +146,7 @@ Then you choose menu item which you want to start.
 
 If you have problems with starting installer.sh, you should try to use `dos2unix installer.sh` or `sed -i 's/\r//' installer.sh` commands.
 
-# DPULSE demos
+# DPULSE demo and use-cases
 
 ### You can start DPULSE and see the main menu on the screen using one of the recommended commands in DPULSE root folder. Don't forget to install all requirements before starting DPULSE
 

apis/api_securitytrails.py

@@ -0,0 +1,59 @@
+import requests
+import sqlite3
+from colorama import Fore, Style
+
+def api_securitytrails_check(domain):
+    conn = sqlite3.connect('apis//api_keys.db')
+    cursor = conn.cursor()
+    cursor.execute("SELECT api_name, api_key FROM api_keys")
+    rows = cursor.fetchall()
+    for row in rows:
+        api_name, api_key = row
+        if api_name == 'SecurityTrails':
+            api_key = str(row[1])
+            print(Fore.GREEN + 'Got SecurityTrails API key. Starting SecurityTrails scan...\n')
+
+    subdomains_url = f"https://api.securitytrails.com/v1/domain/{domain}/subdomains?apikey={api_key}"
+    response = requests.get(subdomains_url)
+
+    url = f"https://api.securitytrails.com/v1/domain/{domain}?apikey={api_key}"
+    general_response = requests.get(url)
+    general_data = general_response.json()
+
+    print(Fore.GREEN + "[DOMAIN GENERAL INFORMATION]\n")
+    print(Fore.GREEN + "Alexa Rank: " + Fore.LIGHTCYAN_EX + f"{general_data['alexa_rank']}")
+    print(Fore.GREEN + "Apex Domain: " + Fore.LIGHTCYAN_EX + f"{general_data['apex_domain']}")
+    print(Fore.GREEN + "Hostname: " + Fore.LIGHTCYAN_EX + f"{general_data['hostname']}" + Style.RESET_ALL)
+
+    print(Fore.GREEN + "\n[DNS RECORDS]" + Style.RESET_ALL)
+    for record_type, record_data in general_data['current_dns'].items():
+        print(Fore.GREEN + f"\n[+] {record_type.upper()} RECORDS:" + Style.RESET_ALL)
+        for value in record_data.get('values', []):
+            if record_type == 'a':
+                print(Fore.GREEN + "IP: " + Fore.LIGHTCYAN_EX + f"{value['ip']} " + Fore.GREEN + "| Organization: " + Fore.LIGHTCYAN_EX + f"{value['ip_organization']}")
+            elif record_type == 'mx':
+                print(Fore.GREEN + "Hostname: " + Fore.LIGHTCYAN_EX + f"{value['hostname']} " + Fore.GREEN + "| Priority: " + Fore.LIGHTCYAN_EX + f"{value['priority']} " + Fore.GREEN +  "| Organization: " + Fore.LIGHTCYAN_EX + f"{value['hostname_organization']}")
+            elif record_type == 'ns':
+                print(Fore.GREEN + "Nameserver: " + Fore.LIGHTCYAN_EX + f"{value['nameserver']} " + Fore.GREEN + "| Organization: " + Fore.LIGHTCYAN_EX + f"{value['nameserver_organization']}")
+            elif record_type == 'soa':
+                print(Fore.GREEN + "Email: " + Fore.LIGHTCYAN_EX + f"{value['email']} " + Fore.GREEN + "| TTL: " + Fore.LIGHTCYAN_EX + f"{value['ttl']}")
+            elif record_type == 'txt':
+                print(Fore.GREEN + "Value: " + Fore.LIGHTCYAN_EX + f"{value['value']}")
+
+    if response.status_code == 200:
+        data = response.json()
+        print(Fore.GREEN + "\n[SUBDOMAINS DEEP ENUMERATION]\n")
+        print(Fore.GREEN + f"Found " + Fore.LIGHTCYAN_EX + f"{data['subdomain_count']} " + Fore.GREEN + "subdomains")
+        print(Fore.GREEN + "Subdomains list: ")
+        for i, subdomain in enumerate(data['subdomains'], start=1):
+            subdomain_url = f"http://{subdomain}.{domain}"
+            try:
+                response = requests.get(subdomain_url, timeout=5)
+                if response.status_code == 200:
+                    print(Fore.GREEN + f"{i}. " + Fore.LIGHTCYAN_EX + f"{subdomain_url} " + Fore.GREEN + "is alive")
+                else:
+                    pass
+            except Exception:
+                pass
+    else:
+        pass

apis/api_virustotal.py

@@ -0,0 +1,49 @@
+import requests
+import sqlite3
+from colorama import Fore, Style
+
+def check_domain(domain, api_key):
+    url = "https://www.virustotal.com/vtapi/v2/domain/report"
+    params = {
+        'domain': domain,
+        'apikey': api_key
+    }
+
+    response = requests.get(url, params=params)
+
+    if response.status_code == 200:
+        return response.json()
+    else:
+        print(f"Error: {response.status_code}")
+        return None
+
+
+def api_virustotal_check(domain):
+    conn = sqlite3.connect('apis//api_keys.db')
+    cursor = conn.cursor()
+    cursor.execute("SELECT api_name, api_key FROM api_keys")
+    rows = cursor.fetchall()
+    for row in rows:
+        api_name, api_key = row
+        if api_name == 'VirusTotal':
+            api_key = str(row[1])
+            print(Fore.GREEN + 'Got VirusTotal API key. Starting VirusTotal scan...\n')
+
+    result = check_domain(domain, api_key)
+
+    if result:
+        print(Fore.GREEN + "[VIRUSTOTAL DOMAIN REPORT]")
+        print(Fore.GREEN + f"Domain: {result.get('domain')}")
+        print(Fore.GREEN + f"Categories: {result.get('categories')}")
+        print(Fore.GREEN + f"Detected URLs: {len(result.get('detected_urls', []))}")
+        print(Fore.GREEN + f"Detected Samples: {len(result.get('detected_samples', []))}")
+        print(Fore.GREEN + f"Undetected Samples: {len(result.get('undetected_samples', []))}\n")
+        print(Fore.LIGHTGREEN_EX + "-------------------------------------------------\n" + Style.RESET_ALL)
+        conn.close()
+    else:
+        print(Fore.RED + "Failed to get domain report\n")
+        print(Fore.LIGHTGREEN_EX + "-------------------------------------------------\n" + Style.RESET_ALL)
+        conn.close()
+        pass
+
+

datagather_modules/crawl_processor.py

@@ -115,25 +115,25 @@ def sm_gather(url):
     for link in links:
         parsed_url = urlparse(link)
         hostname = parsed_url.hostname
-        if hostname and hostname.endswith('facebook.com'):
+        if hostname and (hostname == 'facebook.com' or hostname.endswith('.facebook.com')):
             categorized_links['Facebook'].append(urllib.parse.unquote(link))
-        elif hostname and hostname.endswith('twitter.com'):
+        elif hostname and (hostname == 'twitter.com' or hostname.endswith('.twitter.com')):
             categorized_links['Twitter'].append(urllib.parse.unquote(link))
-        elif hostname and hostname.endswith('instagram.com'):
+        elif hostname and (hostname == 'instagram.com' or hostname.endswith('.instagram.com')):
             categorized_links['Instagram'].append(urllib.parse.unquote(link))
-        elif hostname and hostname.endswith('t.me'):
+        elif hostname and (hostname == 't.me' or hostname.endswith('.t.me')):
             categorized_links['Telegram'].append(urllib.parse.unquote(link))
-        elif hostname and hostname.endswith('tiktok.com'):
+        elif hostname and (hostname == 'tiktok.com' or hostname.endswith('.tiktok.com')):
             categorized_links['TikTok'].append(urllib.parse.unquote(link))
-        elif hostname and hostname.endswith('linkedin.com'):
+        elif hostname and (hostname == 'linkedin.com' or hostname.endswith('.linkedin.com')):
             categorized_links['LinkedIn'].append(urllib.parse.unquote(link))
-        elif hostname and hostname.endswith('vk.com'):
+        elif hostname and (hostname == 'vk.com' or hostname.endswith('.vk.com')):
             categorized_links['VKontakte'].append(urllib.parse.unquote(link))
-        elif hostname and hostname.endswith('youtube.com'):
+        elif hostname and (hostname == 'youtube.com' or hostname.endswith('.youtube.com')):
             categorized_links['YouTube'].append(urllib.parse.unquote(link))
-        elif hostname and hostname.endswith('wechat.com'):
+        elif hostname and (hostname == 'wechat.com' or hostname.endswith('.wechat.com')):
             categorized_links['WeChat'].append(urllib.parse.unquote(link))
-        elif hostname and hostname.endswith('ok.ru'):
+        elif hostname and (hostname == 'ok.ru' or hostname.endswith('.ok.ru')):
             categorized_links['Odnoklassniki'].append(urllib.parse.unquote(link))
 
     if not categorized_links['Odnoklassniki']:
@@ -214,25 +214,25 @@ def domains_reverse_research(subdomains, report_file_type):
     for inner_list in subdomain_socials_grouped:
         for link in inner_list:
             hostname = urlparse(link).hostname
-            if hostname and hostname.endswith('facebook.com'):
+            if hostname and (hostname == 'facebook.com' or hostname.endswith('.facebook.com')):
                 sd_socials['Facebook'].append(urllib.parse.unquote(link))
-            elif hostname and hostname.endswith('twitter.com'):
+            elif hostname and (hostname == 'twitter.com' or hostname.endswith('.twitter.com')):
                 sd_socials['Twitter'].append(urllib.parse.unquote(link))
-            elif hostname and hostname.endswith('instagram.com'):
+            elif hostname and (hostname == 'instagram.com' or hostname.endswith('.instagram.com')):
                 sd_socials['Instagram'].append(urllib.parse.unquote(link))
-            elif hostname and hostname.endswith('t.me'):
+            elif hostname and (hostname == 't.me' or hostname.endswith('.t.me')):
                 sd_socials['Telegram'].append(urllib.parse.unquote(link))
-            elif hostname and hostname.endswith('tiktok.com'):
+            elif hostname and (hostname == 'tiktok.com' or hostname.endswith('.tiktok.com')):
                 sd_socials['TikTok'].append(urllib.parse.unquote(link))
-            elif hostname and hostname.endswith('linkedin.com'):
+            elif hostname and (hostname == 'linkedin.com' or hostname.endswith('.linkedin.com')):
                 sd_socials['LinkedIn'].append(urllib.parse.unquote(link))
-            elif hostname and hostname.endswith('vk.com'):
+            elif hostname and (hostname == 'vk.com' or hostname.endswith('.vk.com')):
                 sd_socials['VKontakte'].append(urllib.parse.unquote(link))
-            elif hostname and hostname.endswith('youtube.com'):
+            elif hostname and (hostname == 'youtube.com' or hostname.endswith('.youtube.com')):
                 sd_socials['YouTube'].append(urllib.parse.unquote(link))
-            elif hostname and hostname.endswith('wechat.com'):
+            elif hostname and (hostname == 'wechat.com' or hostname.endswith('.wechat.com')):
                 sd_socials['WeChat'].append(urllib.parse.unquote(link))
-            elif hostname and hostname.endswith('ok.ru'):
+            elif hostname and (hostname == 'ok.ru' or hostname.endswith('.ok.ru')):
                 sd_socials['Odnoklassniki'].append(urllib.parse.unquote(link))
 
     sd_socials = {k: list(set(v)) for k, v in sd_socials.items()}