README
¶
Identity Matching source{d} Extension
Match different identities of the same person using 🤖. Extension for source{d}.
Overview • How To Use • Science • Contributions • License
Overview
People are using different e-mails and names (aka identities) when they commit their work to git. E-mails can be corporate, personal, special like users.noreply.github.com, etc. Names can be with Surname or without, with typos, no name, etc. Thus to get precise information about developer it is required to gather their identities and separate them from another person identities. That's what we call Identity Matching.
How To Use
Right now no pre-built binaries are available. Please refer to How to build from source code section to build an executable.
Run match-identities --help to see all the parameters that you can configure.
There are two use cases supported for match-identities.
In both cases, the output identity table is saved as a Parquet file. Read more in the Output format section.
Use with gitbase
match-identities is supposed to be used with gitbase.
First of all, make sure you have a gitbase instance running with all the repositories you are going to analyze.
Please refer to the gitbase documentation to get more information.
Usage example:
match-identities --output matched_identities.parquet
The credentials can be configured with the --host, --port, --user and --password flags.
For example, the following SQL gitbase query will return the identities of each commit author:
SELECT DISTINCT repository_id, commit_author_name, commit_author_email
FROM commits;
If you want to cache the gitbase output you can use the --cache flag.
After the identities are fetched from gitbase, the matching process is run.
Read Science section to learn more.
Use without gitbase
If you run match-identities with the --cache option enabled you get a csv file with the cached gitbase output.
Besides, if you already have a list of identities it is possible to run match-identities without gitbase involved.
Create a CSV file with the columns repo, email and name, then feed it to the --cache parameter.
Usage Example:
match-identities \
--cache path/to/csv/file.csv \
--output matched_identities.parquet
Output format
Once the algorithm finishes to merge identities, you get a table with 4 columns:
id(int64) -- unique identifier of the person with the corresponding identity.email(utf8) -- e-mail of the identity.name(utf8) -- name of the identity.repo(utf8) -- repository of the commit.
The columns email, name and repo may contain empty values which means no constraints.
For example, let's consider this output identity table:
id,email,name,repo
1,alice@gmail.com,"",""
1,"",alice,""
2,bob@gmail.com,"",""
2,"",bob,""
2,bob@inbox.com,"",""
2,"",no-name,bob/bobs-project
There are two developers.
Let's name them Alice (with id 1) and Bob (with id 2).
When we analyze a commit with alice@gmail.com as author email, then the author is Alice.
The repository and author name are ignored since the author email is the most reliable way to define an identity.
On the other hand, when we analyze a commit with alice as an author name, then the author is Alice for whatever combination of email and repository.
Same for Bob, although he uses two different email addresses bob@gmail.com and bob@inbox.com.
If we come across a commit with the no-name author name in bob/bobs-project repository then it is Bob's.
Convert parquet to CSV
It is possible to convert the output parquet file to CSV using the python script in the research directory:
python3 ./research/parquet2csv.py matched_identities.parquet
The result will be saved as matched_identities.csv.
Please note that pyspark must be installed.
External matching option
If the organization is using GitHub, Gitlab or Bitbucket, it is possible to use their API to match identities by emails. In that case, 2 columns are added and filled for every email in the table: the External id provider and the External id itself.
How to build
git clone https://github.com/src-d/identity-matching
cd identity-matching
make build
You'll see two directories with Linux and Macos binaries inside the build directory.
Science
There are two stages to match identities. The first is the precomputation which is run once on the whole dataset and remains unchanged during the subsequent steps. The second is the matching itself.
- Precomputation:
- Gather 2 lists of the most popular names and emails (by frequencies) on the whole dataset.
- Gather 2 lists of emails and names that will be ignored (aka blacklists) on the whole dataset. They are non-human identities and usually related to CI, bots, etc.
- Analysis:
- Gather the list of triplets
{email, name, repository}from all the commits using gitbase. - Remove any triplet whose name or email belongs to the blacklists.
- Merge identities with the same e-mail if it doesn't belong to the list of popular emails created in 1.1.
- Merge identities with the same name if it doesn't belong to the list of popular names created in 1.1.
When the name belongs to this list we replace it with the following tuple
(name, repository). - Save the resulting identity table in the desired output format.
- Gather the list of triplets
There is a Design Document (or a Blueprint, or whatever else you are used to call project documentation) which goes into more detail: link.
Contributions
...are welcome! See CONTRIBUTING and code of conduct.
License
GPL 3.0, see LICENSE. Y u no Apache/MIT? Read here.
Documentation
¶
Index ¶
- func Dir(useLocal bool, name string) http.FileSystem
- func FS(useLocal bool) http.FileSystem
- func FSByte(useLocal bool, name string) ([]byte, error)
- func FSMustByte(useLocal bool, name string) []byte
- func FSMustString(useLocal bool, name string) string
- func FSString(useLocal bool, name string) (string, error)
- func HashPeopleDiscoverySQL() string
- func ReducePeople(people People, matcher external.Matcher, blacklist Blacklist, ...) error
- func SetPrimaryValues(people People, nameFreqs, emailFreqs map[string]*Frequency, minRecentCount int)
- type Blacklist
- type Commit
- type Frequency
- type Int64Slice
- type NameWithRepo
- type People
- type Person
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Dir ¶
func Dir(useLocal bool, name string) http.FileSystem
Dir returns a http.Filesystem for the embedded assets on a given prefix dir. If useLocal is true, the filesystem's contents are instead used.
func FS ¶
func FS(useLocal bool) http.FileSystem
FS returns a http.Filesystem for the embedded assets. If useLocal is true, the filesystem's contents are instead used.
func FSByte ¶
FSByte returns the named file from the embedded assets. If useLocal is true, the filesystem's contents are instead used.
func FSMustByte ¶
FSMustByte is the same as FSByte, but panics if name is not present.
func FSMustString ¶
FSMustString is the string version of FSMustByte.
func HashPeopleDiscoverySQL ¶
func HashPeopleDiscoverySQL() string
HashPeopleDiscoverySQL returns the hashsum of the SQL used to fetch the raw Git signatures.
func ReducePeople ¶
func ReducePeople(people People, matcher external.Matcher, blacklist Blacklist, maxIdentities int) error
ReducePeople merges the identities together by following the fixed set of rules.
- Run the external matching, if available.
- Run the series of heuristics on those items which were left untouched in the list (everything in case of ext == nil, not found in case of ext != nil).
The heuristics are: TODO(vmarkovtsev): describe the current approach
func SetPrimaryValues ¶
func SetPrimaryValues(people People, nameFreqs, emailFreqs map[string]*Frequency, minRecentCount int)
SetPrimaryValues sets people primary name and email to the most frequent name and email of the person's identity. Stats for the fixed recent period of time are used if there are at least minRecentCount commits made by the person's identity in that period. Otherwise the stats for all the time are used.
Types ¶
type Blacklist ¶
type Blacklist struct {
Domains map[string]struct{}
TopLevelDomains map[string]struct{}
Names map[string]struct{}
Emails map[string]struct{}
PopularEmails map[string]struct{}
PopularNames map[string]struct{}
}
Blacklist contains all the data to filter identities or identities connection
func NewBlacklist ¶
NewBlacklist generates Blacklist from the data files embedded to blacklists.go
type Frequency ¶
Frequency is a pair of word frequencies for a certain recent period of time and for all the time
type Int64Slice ¶
type Int64Slice []int64
Int64Slice attaches the methods of Interface to []int64, sorting in increasing order.
func (Int64Slice) Len ¶
func (p Int64Slice) Len() int
func (Int64Slice) Less ¶
func (p Int64Slice) Less(i, j int) bool
func (Int64Slice) Swap ¶
func (p Int64Slice) Swap(i, j int)
type NameWithRepo ¶
NameWithRepo is a Name that can be linked to a specific repo.
func (NameWithRepo) String ¶
func (rn NameWithRepo) String() string
String describes the person's identity parts.
type People ¶
People is a map of persons indexed by their ID.
func FindPeople ¶
func FindPeople(ctx context.Context, connString string, cachePath string, blacklist Blacklist, recentMonths int) (People, map[string]*Frequency, map[string]*Frequency, error)
FindPeople returns all the people in the database or from the disk cache.
func (People) ForEach ¶
ForEach executes a function over each person in the collection. The order is fixed and constant.
type Person ¶
type Person struct {
ID int64
NamesWithRepos []NameWithRepo
Emails []string
// SampleCommit in an example Git commit which mentions this identity. May be nil.
SampleCommit *Commit
ExternalID string
PrimaryName string
PrimaryEmail string
}
Person is a single individual that can have multiple names and emails.
Source Files
Directories