Please help me resolve issue.
I have a very large list.
I get the element from generator, that's why i use next() to take elements.
I need to process data by chunks, approximately 100 elements it is a chunk length.
When i got first 100 elements the WHILEloop should wait, until i process this first chunk.
The first half of chunk I write to result deque.
Then i update the buffer until the buffer size will be a 100 elements and do it again.
from collections import deque
BUFFER_SIZE = 100
buffer = deque()
result = deque()
for item in range(1,1000000):
ind2remove = 
for idx,buff in enumerate(buffer):
if buff % 2 == 0:
for i in sorted(ind2remove, reverse=True):
while len(buffer) > BUFFER_SIZE/2:
for buff in buffer:
gen_elem = element_generator()
while True if len(buffer) < BUFFER_SIZE else process_data():
element = gen_elem.next()
print "Result : ", result
I am not sure that it is a most effective and fastest way to do this. Is it any others ideas ? Thanks.