1<h1 align="center">
2TextRank on Go
3</h1>
4
5<p align="center">
6 <a href="https://godoc.org/github.com/DavidBelicza/TextRank">
7 <img src="https://godoc.org/github.com/DavidBelicza/TextRank?status.svg" alt="GoDoc" />
8 </a>
9 <a href="https://github.com/DavidBelicza/TextRank/blob/master/LICENSE">
10 <img src="https://img.shields.io/badge/License-MIT-ee00ee.svg" alt="License: MIT" />
11 </a>
12 <a href="https://travis-ci.org/DavidBelicza/TextRank">
13 <img src="https://travis-ci.org/DavidBelicza/TextRank.svg?branch=master" alt="Build Status" />
14 </a>
15 <a href="https://goreportcard.com/report/github.com/DavidBelicza/TextRank">
16 <img src="https://goreportcard.com/badge/github.com/DavidBelicza/TextRank" alt="Go Report Card" />
17 </a>
18 <a href="https://coveralls.io/github/DavidBelicza/TextRank?branch=master">
19 <img src="https://coveralls.io/repos/github/DavidBelicza/TextRank/badge.svg?branch=master" alt="Coverage Status" />
20 </a>
21 <a href="https://github.com/DavidBelicza/TextRank/releases/latest">
22 <img src="https://img.shields.io/github/release/DavidBelicza/TextRank.svg?colorB=269aca" alt="Release" />
23 </a>
24
25</p>
26
27<p align="center">
28This source code is an implementation of textrank algorithm, under MIT licence.
29<br />The minimum requred Go version is 1.8.
30<p align="center">
31<br />
32
33## MOTIVATION
34
35If there was a program what could rank book size text's words, phrases and sentences continuously on multiple threads and it would be opened to modifing by objects, written in a simple, secure, static language and if it would be very well documented... Now, here it is.
36
37## FEATURES
38
39* Find the most important phrases.
40* Find the most important words.
41* Find the most important N sentences.
42 * Importance by phrase weights.
43 * Importance by word occurrence.
44* Find the first N sentences, start from Xth sentence.
45* Find sentences by phrase chains ordered by position in text.
46* Access to the whole ranked data.
47* Support more languages.
48* Algorithm for weighting can be modified by interface implementation.
49* Parser can be modified by interface implementation.
50* Multi thread support.
51
52## INSTALL
53
54You can install TextRank by Go's get:
55
56```go get github.com/DavidBelicza/TextRank```
57
58TextRank uses the default Go *mod* as vendoring tool, so you can install the dependencies with this command:
59
60```go mod vendor```
61
62## DOCKER
63
64Using Docker to TextRank isn't necessary, it's just an option.
65
66Build image from the repository's root directory:
67
68```docker build -t go_text_rank_image .```
69
70Create container from the image:
71
72```docker run -dit --name textrank go_text_rank_image:latest```
73
74Run the **go test -v .** code inside the container:
75
76```docker exec -i -t textrank go test -v .```
77
78Stop, start or remove the container:
79
80* ```docker stop textrank```
81* ```docker start textrank```
82* ```docker rm textrank```
83
84## HOW DOES IT WORK
85
86Too see how does it work, the easiest way is to use the sample text. Sample text can be found in the [textrank_test.go file at this line](https://github.com/DavidBelicza/TextRank/blob/master/textrank_test.go#L12). It's a short size text about Gnome Shell.
87
88* TextRank reads the text,
89 * parse it,
90 * remove the unnecessary stop words,
91 * tokenize it
92* and counting the occurrence of the words and phrases
93* and then it starts weighting
94 * by the occurrence of words and phrases and their relations.
95* After weights are done, TextRank normalize weights to between 1 and 0.
96* Then the different finder methods capable to find the most important words, phrases or sentences.
97
98The most important phrases from the sample text are:
99
100Phrase | Occurrence | Weight
101--- | --- | ---
102gnome - shell | 5 | 1
103extension - gnome | 3 | 0.50859946
104icons - tray | 3 | 0.49631447
105gnome - caffeine | 2 | 0.27027023
106
107The **gnome** is the most often used word in this text and **shell** is also used multiple times. Two of them are used together as a phrase 5 times. This is the highest occurrence in this text, so this is the most important phrase.
108
109The following two important phrases have same occurrence 3, however they are not equal. This is because the **extension gnome** phrase contains the word **gnome**, the most popular word in the text, and it increases the phrase's weight. It increases the weight of any word what is related to it, but not too much to overcome other important phrases what don't contain the **gnome** word.
110
111The exact algorithm can be found in the [algorithm.go file at this line](https://github.com/DavidBelicza/TextRank/blob/master/rank/algorithm.go#L65).
112
113## TEXTRANK OR AUTOMATIC SUMMARIZATION
114> Automatic summarization is the process of reducing a text document with a computer program in order to create a summary that retains the most important points of the original document. Technologies that can make a coherent summary take into account variables such as length, writing style and syntax. Automatic data summarization is part of machine learning and data mining. The main idea of summarization is to find a representative subset of the data, which contains the information of the entire set. Summarization technologies are used in a large number of sectors in industry today. - Wikipedia
115
116## EXAMPLES
117
118### Find the most important phrases
119
120This is the most basic and simplest usage of textrank.
121
122```go
123package main
124
125import (
126 "fmt"
127
128 "github.com/DavidBelicza/TextRank/v2"
129)
130
131func main() {
132 rawText := "Your long raw text, it could be a book. Lorem ipsum..."
133 // TextRank object
134 tr := textrank.NewTextRank()
135 // Default Rule for parsing.
136 rule := textrank.NewDefaultRule()
137 // Default Language for filtering stop words.
138 language := textrank.NewDefaultLanguage()
139 // Default algorithm for ranking text.
140 algorithmDef := textrank.NewDefaultAlgorithm()
141
142 // Add text.
143 tr.Populate(rawText, language, rule)
144 // Run the ranking.
145 tr.Ranking(algorithmDef)
146
147 // Get all phrases by weight.
148 rankedPhrases := textrank.FindPhrases(tr)
149
150 // Most important phrase.
151 fmt.Println(rankedPhrases[0])
152 // Second important phrase.
153 fmt.Println(rankedPhrases[1])
154}
155```
156
157### All possible pre-defined finder queries
158
159After ranking, the graph contains a lot of valuable data. There are functions in textrank package what contains logic to retrieve those data from the graph.
160
161```go
162package main
163
164import (
165 "fmt"
166
167 "github.com/DavidBelicza/TextRank/v2"
168)
169
170func main() {
171 rawText := "Your long raw text, it could be a book. Lorem ipsum..."
172 // TextRank object
173 tr := textrank.NewTextRank()
174 // Default Rule for parsing.
175 rule := textrank.NewDefaultRule()
176 // Default Language for filtering stop words.
177 language := textrank.NewDefaultLanguage()
178 // Default algorithm for ranking text.
179 algorithmDef := textrank.NewDefaultAlgorithm()
180
181 // Add text.
182 tr.Populate(rawText, language, rule)
183 // Run the ranking.
184 tr.Ranking(algorithmDef)
185
186 // Get all phrases order by weight.
187 rankedPhrases := textrank.FindPhrases(tr)
188 // Most important phrase.
189 fmt.Println(rankedPhrases[0])
190
191 // Get all words order by weight.
192 words := textrank.FindSingleWords(tr)
193 // Most important word.
194 fmt.Println(words[0])
195
196 // Get the most important 10 sentences. Importance by phrase weights.
197 sentences := textrank.FindSentencesByRelationWeight(tr, 10)
198 // Found sentences
199 fmt.Println(sentences)
200
201 // Get the most important 10 sentences. Importance by word occurrence.
202 sentences = textrank.FindSentencesByWordQtyWeight(tr, 10)
203 // Found sentences
204 fmt.Println(sentences)
205
206 // Get the first 10 sentences, start from 5th sentence.
207 sentences = textrank.FindSentencesFrom(tr, 5, 10)
208 // Found sentences
209 fmt.Println(sentences)
210
211 // Get sentences by phrase/word chains order by position in text.
212 sentencesPh := textrank.FindSentencesByPhraseChain(tr, []string{"gnome", "shell", "extension"})
213 // Found sentence.
214 fmt.Println(sentencesPh[0])
215}
216```
217
218### Access to everything
219
220After ranking, the graph contains a lot of valuable data. The GetRank function allows access to the graph and every data can be retrieved from this structure.
221
222```go
223package main
224
225import (
226 "fmt"
227
228 "github.com/DavidBelicza/TextRank/v2"
229)
230
231func main() {
232 rawText := "Your long raw text, it could be a book. Lorem ipsum..."
233 // TextRank object
234 tr := textrank.NewTextRank()
235 // Default Rule for parsing.
236 rule := textrank.NewDefaultRule()
237 // Default Language for filtering stop words.
238 language := textrank.NewDefaultLanguage()
239 // Default algorithm for ranking text.
240 algorithmDef := textrank.NewDefaultAlgorithm()
241
242 // Add text.
243 tr.Populate(rawText, language, rule)
244 // Run the ranking.
245 tr.Ranking(algorithmDef)
246
247 // Get the rank graph.
248 rankData := tr.GetRankData()
249
250 // Get word ID by token/word.
251 wordId := rankData.WordValID["gnome"]
252
253 // Word's weight.
254 fmt.Println(rankData.Words[wordId].Weight)
255 // Word's quantity/occurrence.
256 fmt.Println(rankData.Words[wordId].Qty)
257 // All sentences what contain the this word.
258 fmt.Println(rankData.Words[wordId].SentenceIDs)
259 // All other words what are related to this word on left side.
260 fmt.Println(rankData.Words[wordId].ConnectionLeft)
261 // All other words what are related to this word on right side.
262 fmt.Println(rankData.Words[wordId].ConnectionRight)
263 // The node of this word, it contains the related words and the relation weight.
264 fmt.Println(rankData.Relation.Node[wordId])
265}
266```
267
268### Adding text continuously
269
270It is possibe to add more text after another texts already have been added. The Ranking function can merge these multiple texts and it can recalculate the weights and all related data.
271
272```go
273package main
274
275import (
276 "fmt"
277
278 "github.com/DavidBelicza/TextRank/v2"
279)
280
281func main() {
282 rawText := "Your long raw text, it could be a book. Lorem ipsum..."
283 // TextRank object
284 tr := textrank.NewTextRank()
285 // Default Rule for parsing.
286 rule := textrank.NewDefaultRule()
287 // Default Language for filtering stop words.
288 language := textrank.NewDefaultLanguage()
289 // Default algorithm for ranking text.
290 algorithmDef := textrank.NewDefaultAlgorithm()
291
292 // Add text.
293 tr.Populate(rawText, language, rule)
294 // Run the ranking.
295 tr.Ranking(algorithmDef)
296
297 rawText2 := "Another book or article..."
298 rawText3 := "Third another book or article..."
299
300 // Add text to the previously added text.
301 tr.Populate(rawText2, language, rule)
302 // Add text to the previously added text.
303 tr.Populate(rawText3, language, rule)
304 // Run the ranking to the whole composed text.
305 tr.Ranking(algorithmDef)
306
307 // Get all phrases by weight.
308 rankedPhrases := textrank.FindPhrases(tr)
309
310 // Most important phrase.
311 fmt.Println(rankedPhrases[0])
312 // Second important phrase.
313 fmt.Println(rankedPhrases[1])
314}
315```
316
317### Using different algorithm to ranking text
318
319There are two algorithm has implemented, it is possible to write custom algorithm by Algorithm interface and use it instead of defaults.
320
321```go
322package main
323
324import (
325 "fmt"
326
327 "github.com/DavidBelicza/TextRank/v2"
328)
329
330func main() {
331 rawText := "Your long raw text, it could be a book. Lorem ipsum..."
332 // TextRank object
333 tr := textrank.NewTextRank()
334 // Default Rule for parsing.
335 rule := textrank.NewDefaultRule()
336 // Default Language for filtering stop words.
337 language := textrank.NewDefaultLanguage()
338 // Using a little bit more complex algorithm to ranking text.
339 algorithmChain := textrank.NewChainAlgorithm()
340
341 // Add text.
342 tr.Populate(rawText, language, rule)
343 // Run the ranking.
344 tr.Ranking(algorithmChain)
345
346 // Get all phrases by weight.
347 rankedPhrases := textrank.FindPhrases(tr)
348
349 // Most important phrase.
350 fmt.Println(rankedPhrases[0])
351 // Second important phrase.
352 fmt.Println(rankedPhrases[1])
353}
354```
355
356### Using multiple graphs
357
358Graph ID exists because it is possible run multiple independent text ranking processes.
359
360```go
361package main
362
363import (
364 "fmt"
365
366 "github.com/DavidBelicza/TextRank/v2"
367)
368
369func main() {
370 rawText := "Your long raw text, it could be a book. Lorem ipsum..."
371 // 1th TextRank object
372 tr1 := textrank.NewTextRank()
373 // Default Rule for parsing.
374 rule := textrank.NewDefaultRule()
375 // Default Language for filtering stop words.
376 language := textrank.NewDefaultLanguage()
377 // Default algorithm for ranking text.
378 algorithmDef := textrank.NewDefaultAlgorithm()
379
380 // Add text.
381 tr1.Populate(rawText, language, rule)
382 // Run the ranking.
383 tr1.Ranking(algorithmDef)
384
385 // 2nd TextRank object
386 tr2 := textrank.NewTextRank()
387
388 // Using a little bit more complex algorithm to ranking text.
389 algorithmChain := textrank.NewChainAlgorithm()
390
391 // Add text to the second graph.
392 tr2.Populate(rawText, language, rule)
393 // Run the ranking on the second graph.
394 tr2.Ranking(algorithmChain)
395
396 // Get all phrases by weight from first graph.
397 rankedPhrases := textrank.FindPhrases(tr1)
398
399 // Most important phrase from first graph.
400 fmt.Println(rankedPhrases[0])
401 // Second important phrase from first graph.
402 fmt.Println(rankedPhrases[1])
403
404 // Get all phrases by weight from second graph.
405 rankedPhrases2 := textrank.FindPhrases(tr2)
406
407 // Most important phrase from second graph.
408 fmt.Println(rankedPhrases2[0])
409 // Second important phrase from second graph.
410 fmt.Println(rankedPhrases2[1])
411}
412```
413
414### Using different non-English languages
415
416Engish is used by default but it is possible to add any language. To use other languages a stop word list is required what you can find here: https://github.com/stopwords-iso
417
418```go
419package main
420
421import (
422 "fmt"
423
424 "github.com/DavidBelicza/TextRank/v2"
425)
426
427func main() {
428 rawText := "Your long raw text, it could be a book. Lorem ipsum..."
429 // TextRank object
430 tr := textrank.NewTextRank()
431 // Default Rule for parsing.
432 rule := textrank.NewDefaultRule()
433 // Default Language for filtering stop words.
434 language := textrank.NewDefaultLanguage()
435
436 // Add Spanish stop words (just some example).
437 language.SetWords("es", []string{"uno", "dos", "tres", "yo", "es", "eres"})
438 // Active the Spanish.
439 language.SetActiveLanguage("es")
440
441 // Default algorithm for ranking text.
442 algorithmDef := textrank.NewDefaultAlgorithm()
443
444 // Add text.
445 tr.Populate(rawText, language, rule)
446 // Run the ranking.
447 tr.Ranking(algorithmDef)
448
449 // Get all phrases by weight.
450 rankedPhrases := textrank.FindPhrases(tr)
451
452 // Most important phrase.
453 fmt.Println(rankedPhrases[0])
454 // Second important phrase.
455 fmt.Println(rankedPhrases[1])
456}
457```
458
459### Asynchronous usage by goroutines
460
461It is thread safe. Independent graphs can receive texts in the same time and can be extended by more text also in the same time.
462
463```go
464package main
465
466import (
467 "fmt"
468 "time"
469
470 "github.com/DavidBelicza/TextRank/v2"
471)
472
473func main() {
474 // A flag when program has to stop.
475 stopProgram := false
476 // Channel.
477 stream := make(chan string)
478 // TextRank object.
479 tr := textrank.NewTextRank()
480
481 // Open new thread/routine
482 go func(tr *textrank.TextRank) {
483 // 3 texts.
484 rawTexts := []string{
485 "Very long text...",
486 "Another very long text...",
487 "Second another very long text...",
488 }
489
490 // Add 3 texts to the stream channel, one by one.
491 for _, rawText := range rawTexts {
492 stream <- rawText
493 }
494 }(tr)
495
496 // Open new thread/routine
497 go func() {
498 // Counter how many times texts added to the ranking.
499 i := 1
500
501 for {
502 // Get text from stream channel when it got a new one.
503 rawText := <-stream
504
505 // Default Rule for parsing.
506 rule := textrank.NewDefaultRule()
507 // Default Language for filtering stop words.
508 language := textrank.NewDefaultLanguage()
509 // Default algorithm for ranking text.
510 algorithm := textrank.NewDefaultAlgorithm()
511
512 // Add text.
513 tr.Populate(rawText, language, rule)
514 // Run the ranking.
515 tr.Ranking(algorithm)
516
517 // Set stopProgram flag to true when all 3 text have been added.
518 if i == 3 {
519 stopProgram = true
520 }
521
522 i++
523 }
524 }()
525
526 // The main thread has to run while go-routines run. When stopProgram is
527 // true then the loop has finish.
528 for !stopProgram {
529 time.Sleep(time.Second * 1)
530 }
531
532 // Most important phrase.
533 phrases := textrank.FindPhrases(tr)
534 // Second important phrase.
535 fmt.Println(phrases[0])
536}
537```
538
539## A SIMPLE VISUAL REPRESENTATION
540
541The below image is a representation how works the simplest text ranking algorithm. This algorithm can be replaced by an another one by inject different Algorithm interface implementation.
542
543<img src="https://i.imgur.com/RUdDfBz.jpg" />