Stream a file one line at a time with bufio.Scanner, so memory use stays flat no matter how big the file gets.
os.Open gives a read-only handle. bufio.NewScanner wraps it and hands back one line per Scan call, holding only a small buffer in memory.
Scan returns false at the end of the file and on failure, and those two look identical from the loop. Check scanner.Err afterwards to tell them apart.
f, err := os.Open("access.log")
if err != nil {
// handle error
}
defer f.Close()
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := scanner.Text()
fmt.Println(line)
}
if err := scanner.Err(); err != nil {
// handle error
}When a line is longer than 64KB
A scanner refuses any line over 64KB and stops, which surfaces as a loop that ends early with a bufio.ErrTooLong from Err. Minified JSON on one line hits this often.
Buffer raises the ceiling. The second argument is the new maximum line length.
scanner := bufio.NewScanner(f)
buf := make([]byte, 0, 64*1024)
scanner.Buffer(buf, 10*1024*1024) // allow up to 10MB per line
for scanner.Scan() {
// ...
}bufio.Reader has no line-length ceiling at all. ReadString reads up to and including the delimiter, and returns what it got alongside io.EOF on the final partial line.
Reach for this when a line could be arbitrarily large, or when the file might not end in a newline.
r := bufio.NewReader(f)
for {
line, err := r.ReadString('\n')
if line != "" {
fmt.Print(line)
}
if err == io.EOF {
break
}
if err != nil {
// handle error
}
}Working program
A complete program that writes a small log, then counts the lines mentioning an error.
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
// countErrors reports how many lines of a log file mention an error.
func countErrors(path string) (int, error) {
f, err := os.Open(path)
if err != nil {
return 0, err
}
defer f.Close()
n := 0
scanner := bufio.NewScanner(f)
for scanner.Scan() {
if strings.Contains(scanner.Text(), "ERROR") {
n++
}
}
return n, scanner.Err()
}
func main() {
os.WriteFile("access.log", []byte(
"GET / 200\nGET /x 500 ERROR\nPOST /y 500 ERROR\n"), 0644)
n, err := countErrors("access.log")
if err != nil {
fmt.Println("read failed:", err)
return
}
fmt.Printf("%d error lines\n", n)
}Run it.
$ go run main.go
2 error lines