Published 11 Apr, 2022

Python - How to speed up python instance initialization for millions of objects?

Category Python
Modified : Nov 29, 2022

Python is an interpreted, object-oriented, high-level programming language. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development.

6

I have defined a python class named Edge as follows:

class Edge:
    def __init__(self):
        self.node1 = 0
        self.node2 = 0
        self.weight = 0

Now I have to create approximately 10^6 to 10^7 instances of Edge using:

edges= []
for (i,j,w) in ijw:
    edge = Edge()
    edge.node1 = i
    edge.node2 = j
    edge.weight = w
    edges.append(edge)

I took me approximately 2 seconds in Desktop. Is there any faster way to do?

Answers

There are 3 suggested solutions here and each one has been listed below with a detailed description. The following topics have been covered briefly such as Python, Performance, Instance. These have been categorized in sections for a clear and precise explanation.

55

You can't make it much faster, but I certainly would use __slots__ to save on memory allocations. Also make it possible to pass in the attribute values when creating the instance:

class Edge:
    __slots__ = ('node1', 'node2', 'weight')
    def __init__(self, node1=0, node2=0, weight=0):
        self.node1 = node1
        self.node2 = node2
        self.weight = weight

With the updated __init__ you can use a list comprehension:

edges = [Edge(*args) for args in ijw]

Together these can shave off a decent amount of time creating the objects, roughly halve the time needed.

Comparison creating 1 million objects; the setup:

>>> from random import randrange
>>> ijw = [(randrange(100), randrange(100), randrange(1000)) for _ in range(10 ** 6)]
>>> class OrigEdge:
...     def __init__(self):
...         self.node1 = 0
...         self.node2 = 0
...         self.weight = 0
...
>>> origloop = '''\
... edges= []
... for (i,j,w) in ijw:
...     edge = Edge()
...     edge.node1 = i
...     edge.node2 = j
...     edge.weight = w
...     edges.append(edge)
... '''
>>> class SlotsEdge:
...     __slots__ = ('node1', 'node2', 'weight')
...     def __init__(self, node1=0, node2=0, weight=0):
...         self.node1 = node1
...         self.node2 = node2
...         self.weight = weight
...
>>> listcomploop = '''[Edge(*args) for args in ijw]'''

and the timings:

>>> from timeit import Timer
>>> count, total = Timer(origloop, 'from __main__ import OrigEdge as Edge, ijw').autorange()
>>> (total / count) * 1000 # milliseconds
722.1121070033405
>>> count, total = Timer(listcomploop, 'from __main__ import SlotsEdge as Edge, ijw').autorange()
>>> (total / count) * 1000 # milliseconds
386.6706900007557

That's nearly 2 times as fast.

Increasing the random input list to 10^7 items, and the timing difference holds:

>>> ijw = [(randrange(100), randrange(100), randrange(1000)) for _ in range(10 ** 7)]
>>> count, total = Timer(origloop, 'from __main__ import OrigEdge as Edge, ijw').autorange()
>>> (total / count)
7.183759553998243
>>> count, total = Timer(listcomploop, 'from __main__ import SlotsEdge as Edge, ijw').autorange()
>>> (total / count)
3.8709938440006226

28

There is another fast and memory saving method using recordclass library:

from recordclass import dataobject

from random import randrange
import sys
ijw = [(randrange(100), randrange(100), randrange(1000)) for _ in range(10 ** 7)]

class EdgeDO(dataobject):
    __fields__ = 'node1', 'node2', 'weight'

class EdgeSlots:
    __slots__ = 'node1', 'node2', 'weight'

    def __init__(self, node1, node2, weight):
         self.node1 = node1
         self.node2 = node2
         self.weight = weight
            
def list_size(lst):
    return sum(sys.getsizeof(o) for o in lst)

%time list_do = [EdgeDO(n1, n2, w) for n1, n2, w in ijw]
%time list_slots = [EdgeSlots(n1, n2, w) for n1, n2, w in ijw]

print('size (dataobject):', list_size(list_do))
print('size (__slots__): ', list_size(list_slots))

There is the output:

CPU times: user 2.23 s, sys: 20 ms, total: 2.25 s
Wall time: 2.25 s
CPU times: user 6.79 s, sys: 84.1 ms, total: 6.87 s
Wall time: 6.87 s
size (dataobject): 400000000
size (__slots__):  640000000

P.S. Since 0.15 there is option fast_new for faster instance creation. Below new performance counters for python 3.9, 64 bit.

from random import randrange import sys ijw = [(randrange(100), randrange(100), randrange(1000)) for _ in range(10 ** 7)]

class EdgeDO(dataobject, fast_new=True):
    __fields__ = 'node1', 'node2', 'weight'

class EdgeSlots:
    __slots__ = 'node1', 'node2', 'weight'

    def __init__(self, node1, node2, weight):
         self.node1 = node1
         self.node2 = node2
         self.weight = weight
        
def list_size(lst):
    return sum(sys.getsizeof(o) for o in lst)

print('dataobject timinig:')
%time list_do = [EdgeDO(*args) for args in ijw]
print('__slots__ timinig:')
%time list_slots = [EdgeSlots(*args) for args in ijw]

print('size (dataobject):', list_size(list_do))
print('size (__slots__): ', list_size(list_slots))
print(list_size(list_do)/list_size(list_slots)*100, "%")

Results:

dataobject timinig:
CPU times: user 804 ms, sys: 16 ms, total: 820 ms
Wall time: 819 ms
__slots__ timinig:
CPU times: user 5.54 s, sys: 23.9 ms, total: 5.56 s
Wall time: 5.56 s
size (dataobject): 400000000
size (__slots__):  560000000
71.42857142857143 %

10

Another option is to skip the Edge class and implement the edges via a table, or adjacency matrix.

E.g.

A = create_adjacency_graph(ijw)  # Implement to return a IxJ (sparse?) matrix of weights
edge_a_weight = A[3, 56]
edge_b_weight = A[670, 1023]
# etc...

This does remove some flexibility though, but should be quite fast both to create and use.


Concluding Remarks

If you're new to the programming world, you can absolutely benefit from Python's high level abstractions. It is highly interactive and known for its "strong opinions" around specific syntax. Python like other high-level languages, has a garbage collection process to manage memory or delete unused resources effectively.

These were some of the most noted solutions users voted for. Kindly upvote the solution that was helpful for you and help others.