GfG QA is closing soon... Please refer PRACTICE or GfG now on wards !!

How to process large list by chunks

Please help me resolve issue.
I have a very large list.
I get the element from generator, that's why i use next() to take elements.
I need to process data by chunks, approximately 100 elements it is a chunk length.
When i got first 100 elements the WHILEloop should wait, until i process this first chunk.
The first half of chunk I write to result deque.
Then i update the buffer until the buffer size will be a 100 elements and do it again.

from collections import deque

BUFFER_SIZE = 100
buffer = deque()
result = deque()

def element_generator():
    for item in range(1,1000000):
        yield item

def process_data():
    ind2remove = []
    for idx,buff in enumerate(buffer):
        if buff % 2 == 0:
            ind2remove.append(idx)
    for i in sorted(ind2remove, reverse=True):
        del buffer[i]
    while len(buffer) > BUFFER_SIZE/2:
        result.append(buffer.popleft())
    return True

def proccess_end_in_buffer():
    if buffer:
        for buff in buffer:
            result.append(buff)


gen_elem = element_generator()
while True if len(buffer) < BUFFER_SIZE else process_data():
    try:
        element = gen_elem.next()
        buffer.append(element)
    except StopIteration:
        proccess_end_in_buffer()
        break

print "Result : ", result

I am not sure that it is a most effective and fastest way to do this. Is it any others ideas ? Thanks.

 

asked May 22, 2016 by romensd
...