README ¶
tshark filter
A tshark wrapper tool to filter and/or aggregate packat data in PCAP, generate ECS fields and output them in JSON format. As outputs, currently the followings are supported:
- Stdout
- Elasticsearch
Output JSON
Output JSON includes:
- Filtered packet fields
- Aggregated packet fields
- Elastic Common Schema (ECS) fields are generated.
Example outputs
[
...
{
"@timestamp": "2020-03-22T07:05:22.939Z",
"agent": {
"type": "tshark-filter"
},
"destination": {
"domain": "192.168.1.1",
"ip": "192.168.1.1",
"port": 80
},
"event": {
"created": "2020-03-22T16:28:54.000Z",
"module": "pcap",
"type": "protocol"
},
"file": {
"name": "examples/http2.pcapng"
},
"network": {
"bytes": 441,
"community_id": "1:Zlod455Wgx3PLiShBIQxxxxxxxx=",
"iana_number": "6",
"packets": 1,
"protocol": "http",
"related": {
"ip": [
"192.168.1.1",
"192.168.2.1"
]
},
"transport": "tcp",
"type": "ipv4"
},
"packet": {
"frame_interface_name": "eth0",
"frame_len": "441",
"frame_number": "20",
"frame_protocols": "eth:ethertype:ip:tcp:http",
"frame_time_delta": "0.000132512",
"frame_time_epoch": "1584860722.939166033",
"frame_time_relative": "1.670992815",
"http_host": "192.168.1.1",
"http_request_method": "GET",
"http_request_uri": "/text.exe",
"ip_dst": "192.168.1.1",
"ip_dst_host": "192.168.1.1",
"ip_len": "427",
"ip_proto": "6",
"ip_src": "192.168.2.1",
"ip_src_host": "192.168.2.1",
"ip_version": "4",
"tcp_ack": "1",
"tcp_dstport": "80",
"tcp_flags": "0x00000018",
"tcp_len": "375",
"tcp_nxtseq": "376",
"tcp_seq": "1",
"tcp_srcport": "44208"
},
"source": {
"domain": "192.168.2.1",
"ip": "192.168.2.1",
"port": 44208
}
},
...
]
[
...
{
"@timestamp": "2020-03-06T12:30:57.813Z",
"agent": {
"type": "tshark-filter"
},
"destination": {
"domain": "192.168.0.1",
"ip": "192.168.0.1",
"port": 443
},
"event": {
"created": "2020-03-22T16:20:32.000Z",
"module": "pcap",
"type": "protocol"
},
"file": {
"name": "examples/http1.pcapng"
},
"network": {
"bytes": 251,
"community_id": "1:9Us4XWwcOkDl1xAE6xlxxxxxxxx=",
"iana_number": "6",
"packets": 1,
"protocol": "tls",
"related": {
"ip": [
"192.168.1.1",
"192.168.0.1"
]
},
"transport": "tcp",
"type": "ipv4"
},
"packet": {
"frame_interface_name": "ens33",
"frame_len": "251",
"frame_number": "2422",
"frame_protocols": "eth:ethertype:ip:tcp:tls",
"frame_time_delta": "0.002324240",
"frame_time_epoch": "1583757057.813936138",
"frame_time_relative": "62.307191989",
"ip_dst": "192.168.0.1",
"ip_dst_host": "192.168.0.1",
"ip_len": "237",
"ip_proto": "6",
"ip_src": "192.168.1.1",
"ip_src_host": "192.168.1.1",
"ip_version": "4",
"tcp_ack": "1",
"tcp_dstport": "443",
"tcp_flags": "0x00000018",
"tcp_len": "197",
"tcp_nxtseq": "198",
"tcp_seq": "1",
"tcp_srcport": "49907",
"tls_handshake_extensions_server_name": "login.example.com"
},
"source": {
"domain": "192.168.1.1",
"ip": "192.168.1.1",
"port": 49907
}
},
...
]
Required
Install/Build
- cd $GOPATH/src
- go get -u github.com/adriansr/flowhash
- cd $GOPATH/src/github.com
- mkdir elastic (if needed)
- cd $GOPATH/src/github.com/elastic
- git clone -b 7.x https://github.com/elastic/go-elasticsearch.git
- cd $GOPATH/src/
- git clone https://github.com/rhpenguin/tshark-filter.git
- cd $GOPATH/src/tshark-filter/tshark-filter
- go build
Usage
> tshark-filter --config <config_yml_path> --pcap <pcap_file_path> --action <action_name> --output <output_name> --pretty true
When default_action and/or default_output are specified in a yaml config, --action and/or --output are not needed.
e.g. Stdout output (Filtering)
./tshark-filter --config ./default.yml --pcap http.pcapng --action filter --output stdout --pretty true
e.g. Stdout output (Aggregating packets as a single flow)
./tshark-filter --config ./default.yml --pcap http.pcapng --action agg --output stdout --pretty true
Filtering and/or aggregation rules can be configured in the yaml file.
e.g. Elasticsearch output
./tshark-filter --config ./default.yml --pcap http.pcapng --action filter --output elasticsearch
An Elasticsearch address(url) and auth info can be configured in the yaml file.
Config file in yaml format
- Configurations are saved as a yaml file.
- Filtering and/or aggregation rules are defined.
- Output settings for Elasticsearch are also defined.
e.g. default.yml
#
# Fields names passed to tshark command as -T ek -e <args>.
#
pcap_extracted_fields:
- http.host
- http.request.method
- http.request.uri
- tls.handshake.extensions_server_name
#
# Match conditions for packet fields extracted by pcap_extracted_fields.
#
# The following settings mean "(http_request_method AND http_request_uri) OR tls_handshake_extensions_server_name", for example.
#
pcap_field_conditions:
- http_request_method:
match: regex # 'exact', 'case_ignore', 'exists' or 'regex'
value: ["^P.*","^G.*"] # In case of multiple values, they are ORed.
http_request_uri:
match: exists
- tls_handshake_extensions_server_name:
match: exists
See more examples in tshark-filter/examples directory.
- http.yml : Filtering HTTP fields
- agg_by_community_id.yml : Packets are aggregated as a single flow by using community ID
- elasticsearch.yml : Exporting filtered and aggregated packets to Elasticsearch (by HTTP or Auth/HTTPS)
- download_winexe.yml : Filtering HTTP messages including MS EXE files by using Regex
- CVE_2020_0796.yml : Filtering example for CVE_2020_0796 (a.k.a CoronaBlue or SMBGhost)
- default.yml : Config template
Filtering rules
- exact
- case_ignore
- exists
- regex
Aggregation
Click to show internal directories.
Click to hide internal directories.