gofofa

package module
v0.2.17 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 7, 2024 License: MIT Imports: 20 Imported by: 4

README

gofofa

fofa client in Go

Test status codecov Go Report Card Codacy Badge Github Release Go Reference

Background

The official library doesn't has unittests, 之前官方的库功能不全，代码质量差，完全没有社区活跃度，不符合开源项目的基本要求。因此，想就fofa的客户端作为练手，解决上述问题。

Usage

Build and run
~ go install github.com/LubyRuffy/gofofa/cmd/fofa@latest
~ fofa
~ FOFA_CLIENT_URL='https://fofa.info/?email=xxx@gmail.com&key=xxx' fofa search port=80
  • search query, only query needed:
./fofa search port=80
./fofa search 'port=80 && protocol=ftp'
  • search short, default subcommand is search:
./fofa domain=qq.com
  • custom fields, default 'ip,port':
./fofa search --fields host,ip,port,protocol,lastupdatetime 'port=6379'
./fofa search -f host,ip,port,protocol,lastupdatetime 'port=6379'
  • custom size, default 100:
./fofa search --size 10 'port=6379'
./fofa search -s 10 'port=6379'

if size is larger than your account free limit, you can set -deductMode to decide whether deduct fcoin automatically or not

  • custom out format, default csv: can be csv/json/xml, line by line
./fofa search --format=json 'port=6379'
./fofa search --format json 'port=6379'
  • write to file, default stdout:
./fofa search --outFile a.txt 'port=6379'
./fofa search -o a.txt 'port=6379'
  • fix host to url:
./fofa --size 2 --fields "host" title=Gitblit
47.92.145.232:8998
https://114.55.35.145:8443
./fofa --size 2 --fields "host" --fixUrl title=Gitblit
http://47.92.145.232:8998
https://114.55.35.145:8443
./fofa --size 2 --fields "host" --fixUrl title=Gitblit

use another url prefix:

./fofa --size 1 --fields "host" --fixUrl --urlPrefix "redis://" protocol=redis
redis://152.136.145.87:6379
  • verbose mode
./fofa --verbose search port=80
  • pipeline to nuclei
./fofa -fields "host" -fixUrl 'app="Aspera-Faspex"' | nuclei -t http/cves/2022/CVE-2022-47986.yaml
  • uniq by ip
./fofa --fixUrl --size 1000 --fields host --uniqByIP 'host="edu.cn"'
  • pipeline with parallel mode
fofa -f ip "is_ipv6=false && port=22" | fofa -f ip -uniqByIP -template "port=8443 && ip={}"

can use -rate 3 to increase rate limit, default is 2

Stats
  • stats subcommand
./fofa stats --fields title,country title="hacked by"

fofa stats

Icon
  • icon subcommand

search icon at fofa:

./fofa icon --open ./data/favicon.ico
./fofa icon --open https://fofa.info/favicon.ico
./fofa icon --open http://www.baidu.com

calc local file icon hash:

./fofa icon ./data/favicon.ico

calc remote icon hash:

./fofa icon https://fofa.info/favicon.ico

calc remote homepage icon hash:

./fofa icon http://www.baidu.com
Host
  • host subcommand
./fofa host demo.cpanel.net
Host:            demo.cpanel.net
IP:              208.74.120.133
ASN:             33522
ORG:             CPANEL-INC
Country:         United States of America
CountryCode:     US
Ports:           [2078 3306 2079 2082 143 993 2086 2095 2083 2087 110 2080 80 995 2096 2077 443]
Protocols:       imaps,mysql,https,imap,pop3s,http,pop3
Categories:      Server Management
Products:        cPanel-MGMT-Products
UpdateTime:      2022-05-30 17:00:00
Dump
  • dump large-scale data
./fofa dump --format json -fixUrl -outFile a.json -batchSize 10000 'title=phpinfo'
  • dump large-scale data by queries file (line by line)

cat queries.txt
port=13344
port=23455

# csv
./fofa dump -outFile out.csv -inFile queries.txt

# json
./fofa dump -inFile queries.txt -outFile out.json -j
2023/08/09 10:05:33 dump data of query: port=13344
2023/08/09 10:05:35 size: 11/11, 100.00%
2023/08/09 10:05:35 dump data of query: port=23455
2023/08/09 10:05:37 size: 499/499, 100.00%
Domains
  • domain subcommand 主要用于最简单的拓线

add domains mode to extend domains from domain, through certs

fofa domains -s 1000 -withCount baidu.com
baidu.com       660
dwz.cn  620
dlnel.com       614
bcehost.com     614
bdstatic.com    614
......
......

withCount mean with domain count value, you can also use -uniqByIP to uniq by ip:

fofa domains -s 1000 -withCount -uniqByIP baidu.com 
baidu.com       448
dwz.cn  410
aipage.cn       406
Utils
  • random subcommand

random generate date from fofa, line by line

./fofa random
./fofa random -f host,ip,port,lastupdatetime,title,header,body --format json

every 500ms generate one line, never stop

./fofa random -s -1 -sleep 500
  • count subcommand
./fofa count port=80
  • account subcommand
./fofa account
  • version
./fofa --version

Features

  • ☑ Cross-platform
    • ☑ Windows
    • ☑ Linux
    • ☑ Mac
  • ☑ Code coverage > 90%
  • ☑ As SDK
    • ☑ Client: NewClient
      • ☑ HostSearch
      • ☑ HostSize
      • ☑ AccountInfo
      • ☑ IconHash
      • ☑ support cancel through SetContext
  • ☑ As Client
    • ☑ Sub Commands
      • ☑ account
      • ☑ search
        • ☑ query
        • ☑ fields/f
        • ☑ size/s
        • group/g 根据字段聚合：group by ip 根据ip合并，比如查询一个app会有很多域名，其中多个域名对应一个ip，这时只测试其中一个就好了
        • ☑ fixUrl build valid url，默认的字段如果是http的话前面没有http://前缀，导致命令行一些工具不能使用，通过这个参数进行修复
        • ☑ can use with urlPrefix, such as use app:// instead of http://
        • ☑ support socks5
        • ☑ support redis
        • ☑ full 匹配所有，而不只是一年内的
        • ☑ format
          • ☑ csv
          • ☑ json
          • ☑ xml
          • ☐ table
          • ☐ excel
        • ☑ outFile/o
      • ☑ stats
      • ☑ icon
      • ☐ web
      • ☑ dump https://en.fofa.info/api/batches_pages large-scale data retrieval
      • ☑ domains
    • ☑ Terminal color
    • ☑ Global Config
      • ☑ fofaURL
      • ☑ deductMode
    • ☑ Envirement
      • ☑ FOFA_CLIENT_URL format: /?email=<email>&key=<key>&version=<v1>
      • ☑ FOFA_SERVER
      • ☑ FOFA_EMAIL
      • ☑ FOFA_KEY
  • ☐ Publish
    • ☑ github
    • ☐ brew
    • ☐ apt
    • ☐ yum

Scenes

How to dump all domains that cert is valid and contains google?
./fofa stats -f domain -s 100 'cert.is_valid=true && (cert="google")'

Documentation

Overview

Package gofofa fofa client in Go

env settings: - FOFA_CLIENT_URL full fofa connnection string, format: <url>/?email=<email>&key=<key>&version=<v2> - FOFA_SERVER fofa server - FOFA_EMAIL fofa account email - FOFA_KEY fofa account key

Index

Constants

View Source 
const (
	NoHostWithFixURL = "host field must included when fixUrl option set"
)

Variables

This section is empty.

Functions

func ExtractIconFromHtml 

func ExtractIconFromHtml(data []byte) string

ExtractIconFromHtml extract link icon from html

func FofaURLFromEnv 

func FofaURLFromEnv() string

FofaURLFromEnv parse fofa connection url from env, then generate url string

func IconHash 

func IconHash(iconUrl string) (hash string, err error)

IconHash if url is a local icon file, then calc the hash if url is remote icon url, the download and calc the hash if url is web homepage, then try to parse favicon url and download it, then calc the hash

Types

type AccountInfo 

type AccountInfo struct {
	Error          bool     `json:"error"`            // error or not
	ErrMsg         string   `json:"errmsg,omitempty"` // error string message
	FCoin          int      `json:"fcoin"`            // fcoin count
	FofaPoint      int64    `json:"fofa_point"`       // fofa point
	IsVIP          bool     `json:"isvip"`            // is vip
	VIPLevel       VipLevel `json:"vip_level"`        // vip level
	RemainApiQuery int      `json:"remain_api_query"` // available query
	RemainApiData  int      `json:"remain_api_data"`  // available data amount
}

AccountInfo fofa account info

func (AccountInfo) String 

func (ai AccountInfo) String() string

type Client 

type Client struct {
	Server     string // can set local server for debugging, format: <scheme>://<host>
	APIVersion string // api version
	Email      string // fofa email
	Key        string // fofa key

	Account    AccountInfo // fofa account info
	DeductMode DeductMode  // Deduct Mode
	// contains filtered or unexported fields
}

Client of fofa connection

func NewClient 

func NewClient(options ...ClientOption) (*Client, error)

NewClient from fofa connection string to config and with env config merge

func (*Client) AccountInfo 

func (c *Client) AccountInfo() (ac AccountInfo, err error)

AccountInfo fetch account info from fofa

func (*Client) DumpSearch added in v0.1.12 

func (c *Client) DumpSearch(query string, allSize int, batchSize int, fields []string, onResults func([][]string, int) error, options ...SearchOptions) (err error)

DumpSearch search fofa host data query fofa query string size data size: -1 means all，0 means just data total info, >0 means actual size fields of fofa host search options for search

func (*Client) Fetch 

func (c *Client) Fetch(apiURI string, params map[string]string, v interface{}) (err error)

Fetch http request and parse as json return to v

func (*Client) GetContext added in v0.0.9 

func (c *Client) GetContext() context.Context

GetContext 获取context，用于中止任务

func (*Client) HostSearch 

func (c *Client) HostSearch(query string, size int, fields []string, options ...SearchOptions) (res [][]string, err error)

HostSearch search fofa host data query fofa query string size data size: -1 means all，0 means just data total info, >0 means actual size fields of fofa host search options for search

func (*Client) HostSize 

func (c *Client) HostSize(query string) (count int, err error)

HostSize fetch query matched host count

func (*Client) HostStats 

func (c *Client) HostStats(host string) (data HostStatsData, err error)

HostStats fetch query matched host count

func (*Client) SetContext added in v0.0.9 

func (c *Client) SetContext(ctx context.Context)

SetContext 设置context，用于中止任务

func (*Client) Stats 

func (c *Client) Stats(query string, size int, fields []string) (res []StatsObject, err error)

Stats aggs fofa host data query fofa query string size data size fields' field of fofa host struct

func (*Client) URL 

func (c *Client) URL() string

URL generate fofa connection url string

func (*Client) Update 

func (c *Client) Update(configURL string) error

Update merge config from config url

type ClientOption 

type ClientOption func(c *Client) error

func WithAccountDebug added in v0.1.11 

func WithAccountDebug(v bool) ClientOption

WithAccountDebug 是否错误里面显示账号密码原始信息

func WithLogger 

func WithLogger(logger *logrus.Logger) ClientOption

WithLogger set logger

func WithOnResults added in v0.0.9 

func WithOnResults(onResults func(results [][]string)) ClientOption

WithOnResults set on results callback

func WithURL 

func WithURL(configURL string) ClientOption

WithURL configURL format: <url>/?email=<email>&key=<key>&version=<v2>&tlsdisabled=false&debuglevel=0

type DeductMode 

type DeductMode int

DeductMode should deduct fcoin automatically or just use free limit 

const (
	// DeductModeFree only use free limit size
	DeductModeFree DeductMode = 0
	// DeductModeFCoin deduct fcoin automatically if account has fcoin
	DeductModeFCoin DeductMode = 1
)

func ParseDeductMode 

func ParseDeductMode(v string) DeductMode

ParseDeductMode parse string to DeductMode

type HostResults 

type HostResults struct {
	Mode    string      `json:"mode"`
	Error   bool        `json:"error"`
	Errmsg  string      `json:"errmsg"`
	Query   string      `json:"query"`
	Page    int         `json:"page"`
	Size    int         `json:"size"` // 总数
	Results interface{} `json:"results"`
	Next    string      `json:"next"`
}

HostResults /search/all api results

type HostStatsData 

type HostStatsData struct {
	Error       bool     `json:"error"`
	Errmsg      string   `json:"errmsg"`
	Host        string   `json:"host"`
	IP          string   `json:"ip"`
	ASN         int      `json:"asn"`
	ORG         string   `json:"org"`
	Country     string   `json:"country_name"`
	CountryCode string   `json:"country_code"`
	Protocols   []string `json:"protocol"`
	Ports       []int    `json:"port"`
	Categories  []string `json:"category"`
	Products    []string `json:"product"`
	UpdateTime  string   `json:"update_time"`
}

HostStatsData /host api results

type SearchOptions added in v0.1.11 

type SearchOptions struct {
	FixUrl    bool   // each host fix as url, like 1.1.1.1,80 will change to http://1.1.1.1, https://1.1.1.1:8443 will no change
	UrlPrefix string // default is http://
	Full      bool   // search result for over a year
	UniqByIP  bool   // uniq by ip
}

SearchOptions options of search, for post processors

type StatsItem 

type StatsItem struct {
	Name  string
	Count int
}

StatsItem one stats item

type StatsObject 

type StatsObject struct {
	Name  string
	Items []StatsItem
}

StatsObject one stats object

type StatsResults 

type StatsResults struct {
	Error          bool                   `json:"error"`
	Errmsg         string                 `json:"errmsg"`
	Distinct       map[string]interface{} `json:"distinct"`
	Aggs           map[string]interface{} `json:"aggs"`
	LastUpdateTime string                 `json:"lastupdatetime"`
}

StatsResults /search/stats api results

type VipLevel added in v0.1.12 

type VipLevel int
const (
	VipLevelNone        VipLevel = 0 // 注册用户
	VipLevelNormal      VipLevel = 1 // 普通会员
	VipLevelAdvanced    VipLevel = 2 // 高级会员
	VipLevelEnterprise  VipLevel = 3 // 企业版
	VipLevelEnterprise2 VipLevel = 5 // 企业版
)
const (
	VipLevelSubPersonal VipLevel = 11 // 订阅个人
	VipLevelSubPro      VipLevel = 12 // 订阅专业
	VipLevelSubBuss     VipLevel = 13 // 订阅商业版
)
const (
	VipLevelRed     VipLevel = 20  // 红队版
	VipLevelStudent VipLevel = 22  // 教育账户
	VipLevelNever   VipLevel = 100 // 不可能的等级
)

Source Files

View all Source files

Directories

Path Synopsis
cmd
fofa
fofa/cmd
pkg
outformats

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.