10 Best to Read Files Line by Line in Python

Python has one of the most straightforward file reading and writing implementations. It may be from creating simple text documents to analyzing a server’s internal logs. All of those tasks require essential file reading and writing commands.

In this article, we will be looking at all the ways possible in Python to read a file line by line. Python contains predefined functions for file handling. Therefore, we won’t have any external dependencies.

You try the following implementations on your browser here.

Using Python Read Lines Function

Python readlines() method is a predefined function. Upon calling, it returns us a list type consisting of each line from the document as an element.

Syntax – filename.readlines()
Parameters – hint. This is an optional parameter that mentions the maximum number of bytes to be returned. The default value is -1, which returns all bytes.
Returns – A list type consisting of the file contents as list elements.

Let’s look at the following example.

The following text document will be taken as an example.

myFile = open("PythonPool.txt", "r")
print(myFile.readlines())

Output

['Welcome to PythonPool! Source for Free Python Material.']

Using For Loop to Read Lines

We can use the for loop to read files line by line in Python. The open() function returns an iterable object while opening the document. Using the object, we can iterate over a for loop to read the lines. Let’s take a look at an example.

This is the text file we will be using

myFile = open("PythonPool.txt", "r")
for eachLine in myFile:
  print(eachLine)

Output

Welcome to PythonPool! Source for Free Python Material.

This is line 2

This is line 3

Reading File Lines Using Context Manager

Let’s look at another method in Python to read a file line by line.

In Python, a context manager allows the user to manage files using the “with” keyword alongside File I/O methods. It monitors the file being opened and closes immediately after the file operations finish executing.

In this demonstration, we will be using a context manager along with the For Loop

This is the sample document we will be using:

with open("/content/PythonPool.txt") as myFile:   
    myFileContents = myFile.read()
    print(myFileContents)

Output

Welcome to PythonPool! Source for Free Python Material.
This is line 2
This is line 3

Reading CSV File Line by Line into Array

Using the genfromtxt() feature, we can read data from the file. The function requires 2 parameters:

• File Name
• delimiter – Set of characters is used to separate each line.

from numpy import genfromtxt

myCSV = genfromtxt('/content/sample_data/california_housing_test.csv', delimiter=',')
print(myCSV)

Output

Reading a File Line by Line Backwards

To read a given file backward, we will be making use of the reversed() function. Refer to the following demonstration.

The text document we will be using:

for line in reversed(list(open("/content/PythonPool.txt"))):
    print(line.rstrip())

Output

This is line 3
This is line 2
Welcome to PythonPool! Source for Free Python Material.

However, there is an issue with this implementation. If the entire file cannot be contained within memory, the above program fails.

The alternative solution is to use the file_read_backwards module. This allows us to read large files in memory in a much more operative manner.

from file_read_backwards import FileReadBackwards

with FileReadBackwards("/content/PythonPool.txt", encoding="utf-8") as myFile:
    for eachLine in myFile:
         print(eachLine)

Output

This is line 3
This is line 2
Welcome to PythonPool! Source for Free Python Material.

Reading the File Contents Into a Dictionary

In this scenario, we will be reading the file contents from the following text document into a dictionary.

By creating an empty dictionary, we can enter keys and values from the text file using the .split() function. In order to repeat the process for multiple lines, for loop is used.

prices = {}
with open("/content/SampleText.txt") as f:
    for eachLine in f:
       # splitting between each space character
       (key, value) = eachLine.split() 
       prices[str(key)] = int(value)
print(prices)

Output

{'iPhone': 150000, 'Samsung': 50000, 'OnePlus': 80000, 'Oppo': 35000}

Skipping New Line Spaces While Reading files in Python

Using the context manager, we are able to pass filters on each line and strip off new-line space characters. Refer to the following implementation.

Here is the document in question:

with open("/content/PythonPool.txt") as myFile:
    lines = filter(None, (eachLine.rstrip() for eachLine in myFile))
    for i in lines:
      print(i)

Output

Welcome to PythonPool! Source for Free Python Material.
This is line 2
This is line 3

Reading a CSV File Line by Line using Pandas Module

In very large files, memory may be an issue. Therefore, we may need external dependencies. Let’s see how we can read lines from a file using the Pandas module.

Using the chuncksize parameter of .read_csv(), we are able to read only the aforementioned amount.

Each line is placed into the predefined list.

import pandas as pd
lines=[]
for eachLine in pd.read_csv('/content/sample_data/california_housing_test.csv', encoding='utf-8', header=None, chunksize=1):
    lines.append(eachLine.iloc[0,0])

print(lines)

Output

['longitude', -122.05, -118.3, -117.81, -118.36, -119.67, -119.56, -121.43
...
...
...
...
117.12, -119.63]

Recommended Reading | 6 Ways to Read a CSV file with Numpy in Python

Reading XML File Lines in Python

With the help of the Python XML package, we are able to read XML files line by line.

import xml.etree.ElementTree as ET

tree = ET.parse('/content/myXML.xml')

bookCatalog = tree.findall('book')
print('Number of Books:', len(bookCatalog))

for eachBook in bookCatalog:
    print('Author: ', eachBook.find('author').text)
    print('Title: ', eachBook.find('title').text)
    print('Price: ', eachBook.find('price').text)

Output

Number of Books: 10
Author:  Gambardella, Matthew
Title:  XML Developer's Guide
Price:  44.95

...
...
Author:  O'Brien, Tim
Title:  Microsoft .NET: The Programming Bible
Price:  36.95

Reading JSON Files Line by Line in Python

JSON or JavaScript Object Notation is a format in which data is interchanged. Most APIs send and receive requests in JSON format. Therefore, it is important that we learn how to manage such files within Python.

The following demonstration shows how we can read JSON files line by line.

With the help of the JSON standard library in Python, we are able to work with JSON files.

Here’s the JSON document we will be working with:

import json

data = [ ]
with open("/content/myJSON.txt") as myFile:
    for eachLine in myFIle:
        data.append(json.loads(eachLine))

print(data)

Output

Mango, Large, Yellow

FAQs

Conclusion

We have looked at how we can read files line by line in Python regardless of what format it may be. Various examples for text files, JSON files, CSV files, and XML files have been reviewed.