如何每隔10秒从Go中读取大文件的最后几行
how can I read the last two lines from a big log file without load it into memory completely?
I need read it every 10 secs(On a Win machine)...and I'm stuck trying to read the last lines..
package main
import (
"fmt"
"time"
"os"
)
const MYFILE = "logfile.log"
func main() {
c := time.Tick(10 * time.Second)
for now := range c {
readFile(MYFILE)
}
}
func readFile(fname string){
file, err:=os.Open(fname)
if err!=nil{
panic(err)
}
buf:=make([]byte, 32)
c, err:=file.ReadAt(32, ????)
fmt.Printf("%s
", c)
}
The log file is something like:
07/25/2013 11:55:42.400, 0.559
07/25/2013 11:55:52.200, 0.477
07/25/2013 11:56:02.000, 0.463
07/25/2013 11:56:11.800, 0.454
07/25/2013 11:56:21.600, 0.424
07/25/2013 11:56:31.400, 0.382
07/25/2013 11:56:41.200, 0.353
07/25/2013 11:56:51.000, 0.384
07/25/2013 11:57:00.800, 0.393
07/25/2013 11:57:10.600, 0.456
Thanks!
我如何从大日志文件中读取最后两行而不将其完全加载到内存中? p>
我需要每10秒钟读取一次(在Win机器上)...并且在尝试读取最后几行时我卡住了。 p>
包main
import(
“ fmt”
“ time”
“ os”
)
const MYFILE =“ logfile.log”
func main(){
c:=时间。 暂时勾选(10 * time.Second)
:=范围c {
readFile(MYFILE)
}
}
func readFile(fname string){
file,err:= os.Open( fname)
if err!= nil {
panic(err)
}
buf:= make([[byte,32)
c,err:= file.ReadAt(32,????)\ n fmt.Printf(“%s
”,c)
}
code> pre>
日志文件类似于: p> \ n
2013/07/25 11:55:42.400,0.559
07 / 25/2013 11:55:52.200,0.477
07 / 25/2013 11:56:02.000,0.463
07 / 25/2013 11:56:11.800、0.454
07 / 25/2013 11:56:21.600、0.424
07 / 25/2013 11:56:31.400、0.382
07 / 25/2013 11:56:41.200、0.353 \ n07 / 25/2013 11:56:51.000, 0.384
07 / 25/2013 11:57:00.800,0.393
07 / 25/2013 11:57:10.600,0.456
code> pre>
谢谢! p>
div>
You can use file.Seek() or file.ReadAt() to almost the end and then Reading forward. You can only estimate where to start seeking unless you can know that 2 lines = x bytes.
You can get the File length by using the os.Stat(name)
Here is an example based on ReadAt, Stat, and your sample log file:
package main
import (
"fmt"
"os"
"time"
)
const MYFILE = "logfile.log"
func main() {
c := time.Tick(10 * time.Second)
for _ = range c {
readFile(MYFILE)
}
}
func readFile(fname string) {
file, err := os.Open(fname)
if err != nil {
panic(err)
}
defer file.Close()
buf := make([]byte, 62)
stat, err := os.Stat(fname)
start := stat.Size() - 62
_, err = file.ReadAt(buf, start)
if err == nil {
fmt.Printf("%s
", buf)
}
}
I think a combination of File.Seek(0, 2)
and File.Read()
should work.
The Seek
call gets you to the end of file. You can Seek
to a position a bit before the EOF to get last few lines. Then you Read
till the EOF and just sleep in your goroutine for 10 seconds; next Read
has a chance to get you more data.
You can snatch the idea (and the scan-back logic for initially showing few last lines) from GNU tail
's source.
Well, this is only a raw idea and maybe not the best way, you should check and improve it, but seems to work...
I hope that experienced Go users could contribute too..
With Stat you can get the size of the file and from it get the offset for use with ReadAt
func readLastLine(fname string) {
file, err := os.Open(fname)
if err != nil {
panic(err)
}
defer file.Close()
fi, err := file.Stat()
if err != nil {
fmt.Println(err)
}
buf := make([]byte, 32)
n, err := file.ReadAt(buf, fi.Size()-int64(len(buf)))
if err != nil {
fmt.Println(err)
}
buf = buf[:n]
fmt.Printf("%s", buf)
}
Some people will come to this page looking for efficiently reading the last line of a log file (like the tail command line tool).
Here is my version to read the last line of a big file. It use two previous suggestions (using Seek and file Stat).
It read the file backward, byte by byte (no need to set a buffer size) until finding the beginning of a line or the beginning of the file.
func getLastLineWithSeek(filepath string) string {
fileHandle, err := os.Open(filepath)
if err != nil {
panic("Cannot open file")
os.Exit(1)
}
defer fileHandle.Close()
line := ""
var cursor int64 = 0
stat, _ := fileHandle.Stat()
filesize := stat.Size()
for {
cursor -= 1
fileHandle.Seek(cursor, io.SeekEnd)
char := make([]byte, 1)
fileHandle.Read(char)
if cursor != -1 && (char[0] == 10 || char[0] == 13) { // stop if we find a line
break
}
line = fmt.Sprintf("%s%s", string(char), line) // there is more efficient way
if cursor == -filesize { // stop if we are at the begining
break
}
}
return line
}