1# go-enry [![GoDoc](https://godoc.org/github.com/go-enry/go-enry?status.svg)](https://pkg.go.dev/github.com/go-enry/go-enry/v2) [![Test](https://github.com/go-enry/go-enry/actions/workflows/goTest.yml/badge.svg)](https://github.com/go-enry/go-enry/actions/workflows/goTest.yml?query=branch%3Amaster)
  2
  3Programming language detector and toolbox to ignore binary or vendored files. _enry_, started as a port to _Go_ of the original [Linguist](https://github.com/github/linguist) _Ruby_ library, that has an improved _2x performance_.
  4
  5- [CLI](#cli)
  6- [Library](#library)
  7  - [Use cases](#use-cases)
  8    - [By filename](#by-filename)
  9    - [By text](#by-text)
 10    - [By file](#by-file)
 11    - [Filtering](#filtering-vendoring-binaries-etc)
 12    - [Coloring](#language-colors-and-groups)
 13  - [Languages](#languages)
 14    - [Go](#go)
 15    - [Java bindings](#java-bindings)
 16    - [Python bindings](#python-bindings)
 17    - [Rust bindings](#rust-bindings)
 18- [Divergences from linguist](#divergences-from-linguist)
 19- [Benchmarks](#benchmarks)
 20- [Why Enry?](#why-enry)
 21- [Development](#development)
 22  - [Sync with github/linguist upstream](#sync-with-githublinguist-upstream)
 23- [Misc](#misc)
 24- [License](#license)
 25
 26# CLI
 27
 28The CLI binary is hosted in a separate repository [go-enry/enry](https://github.com/go-enry/enry).
 29
 30# Library
 31
 32_enry_ is also a Go library for guessing a programming language that exposes API through FFI to multiple programming environments.
 33
 34## Use cases
 35
 36_enry_ guesses a programming language using a sequence of matching _strategies_ that are
 37applied progressively to narrow down the possible options. Each _strategy_ varies on the type
 38of input data that it needs to make a decision: file name, extension, the first line of the file, the full content of the file, etc.
 39
 40Depending on available input data, enry API can be roughly divided into the next categories or use cases.
 41
 42### By filename
 43
 44Next functions require only a name of the file to make a guess:
 45
 46- `GetLanguageByExtension` uses only file extension (wich may be ambiguous)
 47- `GetLanguageByFilename` useful for cases like `.gitignore`, `.bashrc`, etc
 48- all [filtering helpers](#filtering)
 49
 50Please note that such guesses are expected not to be very accurate.
 51
 52### By text
 53
 54To make a guess only based on the content of the file or a text snippet, use
 55
 56- `GetLanguageByShebang` reads only the first line of text to identify the [shebang](<https://en.wikipedia.org/wiki/Shebang_(Unix)>).
 57- `GetLanguageByModeline` for cases when Vim/Emacs modeline e.g. `/* vim: set ft=cpp: */` may be present at a head or a tail of the text.
 58- `GetLanguageByClassifier` uses a Bayesian classifier trained on all the `./samples/` from Linguist.
 59
 60  It usually is a last-resort strategy that is used to disambiguate the guess of the previous strategies, and thus it requires a list of "candidate" guesses. One can provide a list of all known languages - keys from the `data.LanguagesLogProbabilities` as possible candidates if more intelligent hypotheses are not available, at the price of possibly suboptimal accuracy.
 61
 62### By file
 63
 64The most accurate guess would be when both a file name and its content are available:
 65
 66- `GetLanguagesByContent` only uses file extension and a set of regexp-based content heuristics.
 67- `GetLanguages` uses the full set of matching strategies and is expected to be most accurate.
 68
 69### Filtering: vendoring, binaries, etc
 70
 71_enry_ expose a set of file-level helpers `Is*` to simplify filtering out the files that are less interesting for the purpose of source code analysis:
 72
 73- `IsBinary`
 74- `IsVendor`
 75- `IsConfiguration`
 76- `IsDocumentation`
 77- `IsDotFile`
 78- `IsImage`
 79- `IsTest`
 80- `IsGenerated`
 81
 82### Language colors and groups
 83
 84_enry_ exposes function to get language color to use for example in presenting statistics in graphs:
 85
 86- `GetColor`
 87- `GetLanguageGroup` can be used to group similar languages together e.g. for `Less` this function will return `CSS`
 88
 89## Languages
 90
 91### Go
 92
 93In a [Go module](https://github.com/golang/go/wiki/Modules),
 94import `enry` to the module by running:
 95
 96```sh
 97go get github.com/go-enry/go-enry/v2
 98```
 99
100The rest of the examples will assume you have either done this or fetched the
101library into your `GOPATH`.
102
103```go
104// The examples here and below assume you have imported the library.
105import "github.com/go-enry/go-enry/v2"
106
107lang, safe := enry.GetLanguageByExtension("foo.go")
108fmt.Println(lang, safe)
109// result: Go true
110
111lang, safe := enry.GetLanguageByContent("foo.m", []byte("<matlab-code>"))
112fmt.Println(lang, safe)
113// result: Matlab true
114
115lang, safe := enry.GetLanguageByContent("bar.m", []byte("<objective-c-code>"))
116fmt.Println(lang, safe)
117// result: Objective-C true
118
119// all strategies together
120lang := enry.GetLanguage("foo.cpp", []byte("<cpp-code>"))
121// result: C++ true
122```
123
124Note that the returned boolean value `safe` is `true` if there is only one possible language detected.
125
126A plural version of the same API allows getting a list of all possible languages for a given file.
127
128```go
129langs := enry.GetLanguages("foo.h",  []byte("<cpp-code>"))
130// result: []string{"C", "C++", "Objective-C}
131
132langs := enry.GetLanguagesByExtension("foo.asc", []byte("<content>"), nil)
133// result: []string{"AGS Script", "AsciiDoc", "Public Key"}
134
135langs := enry.GetLanguagesByFilename("Gemfile", []byte("<content>"), []string{})
136// result: []string{"Ruby"}
137```
138
139### Java bindings
140
141Generated Java bindings using a C shared library and JNI are available under [`java`](https://github.com/go-enry/go-enry/blob/master/java).
142
143A library is published on Maven as [tech.sourced:enry-java](https://mvnrepository.com/artifact/tech.sourced/enry-java) for macOS and linux platforms. Windows support is planned under [src-d/enry#150](https://github.com/src-d/enry/issues/150).
144
145### Python bindings
146
147Python bindings using a C shared library and cffi are available for Python 3.9+.
148
149**Installation:**
150```bash
151pip install enry
152```
153
154Pre-built wheels are available for Linux (x86_64) and macOS (Intel + Apple Silicon).
155
156**Usage:**
157```python
158import enry
159
160language = enry.get_language("example.go", b"package main")
161print(language)  # Output: Go
162```
163
164A library is published on pypi as [enry](https://pypi.org/project/enry/) for
165macOS and linux platforms.
166
167For more details, see [python/README.md](https://github.com/go-enry/go-enry/blob/master/python/README.md).
168
169
170### Rust bindings
171
172Generated Rust bindings using a C static library are available at https://github.com/go-enry/rs-enry.
173
174
175## Divergences from Linguist
176
177The `enry` library is based on the data from `github/linguist` version **v9.5.0**.
178
179Parsing [linguist/samples](https://github.com/github/linguist/tree/master/samples) the following `enry` results are different from the Linguist:
180
181- [Heuristics for ".txt" extension](https://github.com/github/linguist/blob/8083cb5a89cee2d99f5a988f165994d0243f0d1e/lib/linguist/heuristics.yml#L521) in Vim Help File could not be parsed, due to unsupported negative lookahead in RE2 regexp engine.
182
183- [Heuristics for ".sol" extension](https://github.com/github/linguist/blob/8083cb5a89cee2d99f5a988f165994d0243f0d1e/lib/linguist/heuristics.yml#L464) in Solidity could not be parsed, due to unsupported negative lookahead in RE2 regexp engine.
184
185- [Heuristics for ".rno" extension](https://github.com/github/linguist/blob/3a1bd3c3d3e741a8aaec4704f782e06f5cd2a00d/lib/linguist/heuristics.yml#L365) in RUNOFF could not be parsed, due to unsupported lookahead in RE2 regexp engine.
186
187- [Heuristics for ".inc" extension](https://github.com/github/linguist/blob/f0e2d0d7f1ce600b2a5acccaef6b149c87d8b99c/lib/linguist/heuristics.yml#L222) in NASL could not be parsed, due to unsupported possessive quantifier in RE2 regexp engine.
188
189- [Heuristics for ".as" extension](https://github.com/github/linguist/blob/223c00bb80eff04788e29010f98c5778993d2b2a/lib/linguist/heuristics.yml#L67) in ActionScript could not be parsed, due to unsupported positive lookahead in RE2 regexp engine.
190
191- [Heuristics for ".csc", ".gsc" and ".gsh" extension](https://github.com/github/linguist/blob/7469c7982d93f2ad922230d712f586a353dc1a42/lib/linguist/heuristics.yml#L650-L651) in GSC could not be parsed, due to unsupported non-backtracking subexpressions in RE2 regexp engine.
192
193- [Heuristic for ".txt"](https://github.com/github/linguist/blob/bf853f1c663903e3ee35935189760191f1c45e1c/lib/linguist/heuristics.yml#L680-L702) detecting 'Adblock Filter List' regexp syntax not supported by RE2
194
195- [IsVendor('bootstrap.css') == false](https://github.com/github/linguist/blob/v7.23.0/lib/linguist/vendor.yml#L77) v7.23 first unsupported RE syntax outside content heuristics
196
197- As of [Linguist v5.3.2](https://github.com/github/linguist/releases/tag/v5.3.2) it is using [flex-based scanner in C for tokenization](https://github.com/github/linguist/pull/3846). Enry still uses [extract_token](https://github.com/github/linguist/pull/3846/files#diff-d5179df0b71620e3fac4535cd1368d15L60) regex-based algorithm. See [#193](https://github.com/src-d/enry/issues/193).
198
199- Bayesian classifier can't distinguish "SQL" from "PLpgSQL. See [#194](https://github.com/src-d/enry/issues/194).
200
201- Overriding languages and types though `.gitattributes` is not yet supported. See [#18](https://github.com/src-d/enry/issues/18).
202
203- `enry` CLI output does NOT exclude `.gitignore`ed files and git submodules, as Linguist does
204
205In all the cases above that have an issue number - we plan to update enry to match Linguist behavior.
206
207> All the issues related to heuristics' regexp  syntax incompatibilities with the RE2 engine can be avoided by using `oniguruma` instead (see [instuctions](#misc))
208
209## Benchmarks
210
211Enry's language detection has been compared with Linguist's on [_linguist/samples_](https://github.com/github/linguist/tree/master/samples).
212
213We got these results:
214
215![histogram](benchmarks/histogram/distribution.png)
216
217The histogram shows the _number of files_ (y-axis) per _time interval bucket_ (x-axis).
218Most of the files were detected faster by enry.
219
220There are several cases where enry is slower than Linguist due to
221Go regexp engine being slower than Ruby's on, wich is based on [oniguruma](https://github.com/kkos/oniguruma) library, written in C.
222
223See [instructions](#misc) for running enry with oniguruma.
224
225## Why Enry?
226
227In the movie [My Fair Lady](https://en.wikipedia.org/wiki/My_Fair_Lady), [Professor Henry Higgins](http://www.imdb.com/character/ch0011719/) is a linguist who at the very beginning of the movie enjoys guessing the origin of people based on their accent.
228
229"Enry Iggins" is how [Eliza Doolittle](http://www.imdb.com/character/ch0011720/), [pronounces](https://www.youtube.com/watch?v=pwNKyTktDIE) the name of the Professor.
230
231## Development
232
233To run the tests use:
234
235    go test ./...
236
237Setting `ENRY_TEST_REPO` to a path to the existing checkout of the Linguist will avoid cloning it and speeds tests up.
238Setting `ENRY_DEBUG=1` will provide insight into the Bayesian classifier built during `make code-generate`.
239
240### Sync with github/linguist upstream
241
242_enry_ re-uses parts of the original [github/linguist](https://github.com/github/linguist) to generate internal data structures.
243In order to update to the latest release of linguist do:
244
245```bash
246$ git clone https://github.com/github/linguist.git .linguist
247$ cd .linguist; git checkout <release-tag>; cd ..
248
249# put the new release's commit sha in the generator_test.go (to re-generate .gold test fixtures)
250# https://github.com/go-enry/go-enry/blob/13d3d66d37a87f23a013246a1b0678c9ee3d524b/internal/code-generator/generator/generator_test.go#L18
251
252$ make code-generate
253```
254
255To stay in sync, enry needs to be updated when a new release of the linguist includes changes to any of the following files:
256
257- [languages.yml](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml)
258- [heuristics.yml](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.yml)
259- [vendor.yml](https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml)
260- [documentation.yml](https://github.com/github/linguist/blob/master/lib/linguist/documentation.yml)
261
262There now is automation for detecting the changes in the upstream Linguist project: every day Github CI runs [a job](.github/workflows/sync-linguist.yml) that will create a PR to this repo for each new Linguist release. It will include all the steps from the above.
263
264When submitting a pull request syncing up to a new release manually, please make sure it only contains the changes in
265the generated files (in [data](https://github.com/go-enry/go-enry/blob/master/data) subdirectory).
266
267Separating all the necessary "manual" code changes to a different PR that includes some background description and an update to the documentation on ["divergences from linguist"](#divergences-from-linguist) is encouraged and very much appreciated, as it simplifies the maintenance (review/release notes/etc).
268
269## Misc
270
271<details>
272  <summary>Running a benchmark & faster regexp engine</summary>
273
274### Benchmark
275
276All benchmark scripts are in [_benchmarks_](https://github.com/go-enry/go-enry/blob/master/benchmarks) directory.
277
278#### Dependencies
279
280As benchmarks depend on Ruby and GitHub-Linguist gem make sure you have:
281
282- Ruby (e.g using [`rbenv`](https://github.com/rbenv/rbenv)), [`bundler`](https://bundler.io/) installed
283- Docker
284- [native dependencies](https://github.com/github/linguist/#dependencies) installed
285- Build the gem `cd .linguist && bundle install && rake build_gem && cd -`
286- Install it `gem install --no-rdoc --no-ri --local .linguist/github-linguist-*.gem`
287
288#### Quick benchmark
289
290To run quicker benchmarks
291
292    make benchmarks
293
294to get average times for the primary detection function and strategies for the whole samples set. If you want to see measures per sample file use:
295
296    make benchmarks-samples
297
298#### Full benchmark
299
300If you want to reproduce the same benchmarks as reported above:
301
302- Make sure all [dependencies](#benchmark-dependencies) are installed
303- Install [gnuplot](http://gnuplot.info) (in order to plot the histogram)
304- Run `ENRY_TEST_REPO="$PWD/.linguist" benchmarks/run.sh` (takes ~15h)
305
306It will run the benchmarks for enry and Linguist, parse the output, create csv files and plot the histogram.
307
308### Faster regexp engine (optional)
309
310[Oniguruma](https://github.com/kkos/oniguruma) is CRuby's regular expression engine.
311It is very fast and performs better than the one built into Go runtime. _enry_ supports swapping
312between those two engines thanks to [rubex](https://github.com/moovweb/rubex) project.
313The typical overall speedup from using Oniguruma is 1.5-2x. However, it requires CGo and the external shared library.
314On macOS with [Homebrew](https://brew.sh/), it is:
315
316```
317brew install oniguruma
318```
319
320On Ubuntu, it is
321
322```
323sudo apt install libonig-dev
324```
325
326To build enry with Oniguruma regexps use the `oniguruma` build tag
327
328```
329go get -v -t --tags oniguruma ./...
330```
331
332and then rebuild the project.
333
334</details>
335
336## License
337
338Apache License, Version 2.0. See [LICENSE](LICENSE)