Unexpected behaviour with Python generator
I was running a piece of code that unexpectedly gave a logic error at one part of the program. When investigating the section, I created a test file to test the set of statements being run and found out an unusual bug that seems very odd.
I tested this simple code:
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original to something else
print(list(f)) # Outputs filtered
And the output was:
>>>
Yes, nothing. I was expecting the filter comprehension to get items in the array with a count of 2 and output this, but I didn't get that:
# Expected output
>>> [2, 2]
When I commented out the third line to test it once again:
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
### array = [5, 6, 1, 2, 9] # Ignore line
print(list(f)) # Outputs filtered
The output was correct (you can test it for yourself):
>>> [2, 2]
At one point I outputted the type of the variable 'f':
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original
print(type(f))
print(list(f)) # Outputs filtered
And I got:
>>> <class 'generator'>
>>>
TL;DR: Why is updating a list in Python changing the output of another generator variable? This seems very odd to me.
python
|
show 1 more comment
I was running a piece of code that unexpectedly gave a logic error at one part of the program. When investigating the section, I created a test file to test the set of statements being run and found out an unusual bug that seems very odd.
I tested this simple code:
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original to something else
print(list(f)) # Outputs filtered
And the output was:
>>>
Yes, nothing. I was expecting the filter comprehension to get items in the array with a count of 2 and output this, but I didn't get that:
# Expected output
>>> [2, 2]
When I commented out the third line to test it once again:
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
### array = [5, 6, 1, 2, 9] # Ignore line
print(list(f)) # Outputs filtered
The output was correct (you can test it for yourself):
>>> [2, 2]
At one point I outputted the type of the variable 'f':
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original
print(type(f))
print(list(f)) # Outputs filtered
And I got:
>>> <class 'generator'>
>>>
TL;DR: Why is updating a list in Python changing the output of another generator variable? This seems very odd to me.
python
2
You keep printinglist(g)
, butg
is an undefined symbol. Perhaps you have a typo, usingg
forf
?
– Prune
8 hours ago
Yh that is what I meant sorry
– Suraj Kothari
8 hours ago
3
You redefinearray
and your new array is what gets referenced by the lazy generator comprehension.
– jpp
8 hours ago
Would be good to see an answer that mentions scope.
– trailing_whitespace
1 hour ago
1
Slap "strange" onto the title and you're sure to get a few upvotes.
– coldspeed
1 hour ago
|
show 1 more comment
I was running a piece of code that unexpectedly gave a logic error at one part of the program. When investigating the section, I created a test file to test the set of statements being run and found out an unusual bug that seems very odd.
I tested this simple code:
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original to something else
print(list(f)) # Outputs filtered
And the output was:
>>>
Yes, nothing. I was expecting the filter comprehension to get items in the array with a count of 2 and output this, but I didn't get that:
# Expected output
>>> [2, 2]
When I commented out the third line to test it once again:
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
### array = [5, 6, 1, 2, 9] # Ignore line
print(list(f)) # Outputs filtered
The output was correct (you can test it for yourself):
>>> [2, 2]
At one point I outputted the type of the variable 'f':
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original
print(type(f))
print(list(f)) # Outputs filtered
And I got:
>>> <class 'generator'>
>>>
TL;DR: Why is updating a list in Python changing the output of another generator variable? This seems very odd to me.
python
I was running a piece of code that unexpectedly gave a logic error at one part of the program. When investigating the section, I created a test file to test the set of statements being run and found out an unusual bug that seems very odd.
I tested this simple code:
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original to something else
print(list(f)) # Outputs filtered
And the output was:
>>>
Yes, nothing. I was expecting the filter comprehension to get items in the array with a count of 2 and output this, but I didn't get that:
# Expected output
>>> [2, 2]
When I commented out the third line to test it once again:
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
### array = [5, 6, 1, 2, 9] # Ignore line
print(list(f)) # Outputs filtered
The output was correct (you can test it for yourself):
>>> [2, 2]
At one point I outputted the type of the variable 'f':
array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original
print(type(f))
print(list(f)) # Outputs filtered
And I got:
>>> <class 'generator'>
>>>
TL;DR: Why is updating a list in Python changing the output of another generator variable? This seems very odd to me.
python
python
edited 48 mins ago
Ian Kemp
16.5k126798
16.5k126798
asked 8 hours ago
Suraj KothariSuraj Kothari
471213
471213
2
You keep printinglist(g)
, butg
is an undefined symbol. Perhaps you have a typo, usingg
forf
?
– Prune
8 hours ago
Yh that is what I meant sorry
– Suraj Kothari
8 hours ago
3
You redefinearray
and your new array is what gets referenced by the lazy generator comprehension.
– jpp
8 hours ago
Would be good to see an answer that mentions scope.
– trailing_whitespace
1 hour ago
1
Slap "strange" onto the title and you're sure to get a few upvotes.
– coldspeed
1 hour ago
|
show 1 more comment
2
You keep printinglist(g)
, butg
is an undefined symbol. Perhaps you have a typo, usingg
forf
?
– Prune
8 hours ago
Yh that is what I meant sorry
– Suraj Kothari
8 hours ago
3
You redefinearray
and your new array is what gets referenced by the lazy generator comprehension.
– jpp
8 hours ago
Would be good to see an answer that mentions scope.
– trailing_whitespace
1 hour ago
1
Slap "strange" onto the title and you're sure to get a few upvotes.
– coldspeed
1 hour ago
2
2
You keep printing
list(g)
, but g
is an undefined symbol. Perhaps you have a typo, using g
for f
?– Prune
8 hours ago
You keep printing
list(g)
, but g
is an undefined symbol. Perhaps you have a typo, using g
for f
?– Prune
8 hours ago
Yh that is what I meant sorry
– Suraj Kothari
8 hours ago
Yh that is what I meant sorry
– Suraj Kothari
8 hours ago
3
3
You redefine
array
and your new array is what gets referenced by the lazy generator comprehension.– jpp
8 hours ago
You redefine
array
and your new array is what gets referenced by the lazy generator comprehension.– jpp
8 hours ago
Would be good to see an answer that mentions scope.
– trailing_whitespace
1 hour ago
Would be good to see an answer that mentions scope.
– trailing_whitespace
1 hour ago
1
1
Slap "strange" onto the title and you're sure to get a few upvotes.
– coldspeed
1 hour ago
Slap "strange" onto the title and you're sure to get a few upvotes.
– coldspeed
1 hour ago
|
show 1 more comment
7 Answers
7
active
oldest
votes
As others have mentioned Python generators are lazy. When this line is run:
f = (x for x in array if array.count(x) == 2) # Filters original
nothing actually happens yet. You've just declared how the generator function f will work. Array is not looked at yet. Then, you create a new array that replaces the first one, and finally when you call
print(list(f)) # Outputs filtered
the generator now needs the actual values and starts pulling them from the generator f. But at this point, array already refers to the second one, so you get an empty list.
If you need to reassign the list, and can't use a different variable to hold it, consider creating the list instead of a generator in the second line:
f = [x for x in array if array.count(x) == 2] # Filters original
...
print(f)
add a comment |
Generators are lazy, they won't be evaluated until you iterate through them. In this case that's at the point you create the list
with the generator as input, at the print
.
When am I iterating through them. Am I meant to?
– Suraj Kothari
8 hours ago
@SurajKothari when you create thelist
it will iterate for you without you needing to do it explicitly.
– Mark Ransom
8 hours ago
Also which list? When I declare the first one, or re-assign the second?
– Suraj Kothari
8 hours ago
What first & second? You define only one list, at the final line of your code.
– Prune
8 hours ago
@SurajKothari I updated the answer to be more clear.
– Mark Ransom
8 hours ago
add a comment |
You are not using a generator correctly if this is the primary use of this code. Use a list comprehension instead of a generator comprehension. Just replace the parentheses with brackets. It evaluates to a list if you don't know.
array = [1, 2, 2, 4, 5]
f = [x for x in array if array.count(x) == 2]
array = [5, 6, 1, 2, 9]
print(f)
#[2, 2]
You are getting this response because of the nature of a generator. You're calling the generator when it't contents will evaluate to
Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.
– Suraj Kothari
8 hours ago
With your change,list(f)
becomes redundant.
– Mark Ransom
8 hours ago
Lol @Mark Ransom, copy paste got me, I edited.
– Jaba
8 hours ago
@SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!
– Jaba
8 hours ago
add a comment |
Generator evaluation is "lazy" -- it doesn't get executed until you actualize it with a proper reference. With your line:
Look again at your output with the type of f
: that object is a generator, not a sequence. It's waiting to be used, an iterator of sorts.
Your generator isn't evaluated until you start requiring values from it. At that point, it uses the available values at that point, not the point at which it was defined.
Code to "make it work"
That depends on what you mean by "make it work". If you want f
to be a filtered list, then use a list, not a generator:
f = [x for x in array if array.count(x) == 2] # Filters original
I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.
– Suraj Kothari
8 hours ago
add a comment |
Generators are lazy and your newly defined array
is used when you exhaust your generator after redefining. Therefore, the output is correct. A quick fix is to use a list comprehension by replacing parentheses ()
by brackets .
Moving on to how better to write your logic, counting a value in a loop has quadratic complexity. For an algorithm that works in linear time, you can use collections.Counter
to count values, and keep a copy of your original list:
from collections import Counter
array = [1, 2, 2, 4, 5] # original array
counts = Counter(array) # count each value in array
old_array = array.copy() # make copy
array = [5, 6, 1, 2, 9] # updates array
# order relevant
res = [x for x in old_array if counts[x] >= 2]
print(res)
# [2, 2]
# order irrelevant
from itertools import chain
res = list(chain.from_iterable([x]*count for x, count in counts.items() if count >= 2))
print(res)
# [2, 2]
Notice the second version doesn't even require old_array
and is useful if there is no need to maintain ordering of values in your original array.
add a comment |
The root cause of the problem is that generators are lazy; variables are evaluated each time:
>>> l = [1, 2, 2, 4, 5, 5, 5]
>>> filtered = (x for x in l if l.count(x) == 2)
>>> l = [1, 2, 4, 4, 5, 6, 6]
>>> list(filtered)
[4]
It iterates over the original list and evaluates the condition with the current list. In this case, 4 appeared twice in the new list, causing it to appear in the result. It only appears once in the result because it only appeared once in the original list. The 6s appear twice in the new list, but never appear in the old list and are hence never shown.
Full function introspection for the curious (the line with the comment is the important line):
>>> l = [1, 2, 2, 4, 5]
>>> filtered = (x for x in l if l.count(x) == 2)
>>> l = [1, 2, 4, 4, 5, 6, 6]
>>> list(filtered)
[4]
>>> def f(original, new, count):
current = original
filtered = (x for x in current if current.count(x) == count)
current = new
return list(filtered)
>>> from dis import dis
>>> dis(f)
2 0 LOAD_FAST 0 (original)
3 STORE_DEREF 1 (current)
3 6 LOAD_CLOSURE 0 (count)
9 LOAD_CLOSURE 1 (current)
12 BUILD_TUPLE 2
15 LOAD_CONST 1 (<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>)
18 LOAD_CONST 2 ('f.<locals>.<genexpr>')
21 MAKE_CLOSURE 0
24 LOAD_DEREF 1 (current)
27 GET_ITER
28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
31 STORE_FAST 3 (filtered)
4 34 LOAD_FAST 1 (new)
37 STORE_DEREF 1 (current)
5 40 LOAD_GLOBAL 0 (list)
43 LOAD_FAST 3 (filtered)
46 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
49 RETURN_VALUE
>>> f.__code__.co_varnames
('original', 'new', 'count', 'filtered')
>>> f.__code__.co_cellvars
('count', 'current')
>>> f.__code__.co_consts
(None, <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>, 'f.<locals>.<genexpr>')
>>> f.__code__.co_consts[1]
<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>
>>> dis(f.__code__.co_consts[1])
3 0 LOAD_FAST 0 (.0)
>> 3 FOR_ITER 32 (to 38)
6 STORE_FAST 1 (x)
9 LOAD_DEREF 1 (current) # This loads the current list every time, as opposed to loading a constant.
12 LOAD_ATTR 0 (count)
15 LOAD_FAST 1 (x)
18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
21 LOAD_DEREF 0 (count)
24 COMPARE_OP 2 (==)
27 POP_JUMP_IF_FALSE 3
30 LOAD_FAST 1 (x)
33 YIELD_VALUE
34 POP_TOP
35 JUMP_ABSOLUTE 3
>> 38 LOAD_CONST 0 (None)
41 RETURN_VALUE
>>> f.__code__.co_consts[1].co_consts
(None,)
To reiterate: The list to be iterated is only loaded once. Any closures in the condition or expression, however, are loaded from the enclosing scope each iteration. They are not stored in a constant.
The best solution for your problem would be to create a new variable referencing the original list and use that in your generator expression,.
add a comment |
Others have already explained the root cause of the issue - the generator is binding to the name of the array
local variable, rather than its value.
The most pythonic solution is definitely the list comprehension:
f = [x for x in array if array.count(x) == 2]
However, if there is some reason that you don't want to create a list, you can also force a scope close over array
:
f = (lambda array=array: (x for x in array if array.count(x) == 2))()
What's happening here is that the lambda
captures the reference to array
at the time the line is run, ensuring that the generator sees the variable you expect, even if the variable is later redefined.
Note that this still binds to the variable (reference), not the value, so, for example, the following will print [2, 2, 4, 4]
:
array = [1, 2, 2, 4, 5] # Original array
f = (lambda array=array: (x for x in array if array.count(x) == 2))() # Close over array
array.append(4) # This *will* be captured
array = [5, 6, 1, 2, 9] # Updates original to something else
print(list(f)) # Outputs [2, 2, 4, 4]
This is a common pattern in some languages, but it's not very pythonic, so only really makes sense if there's a very good reason for not using the list comprehension (e.g., if array
is very long, or is being used in a nested generator comprehension, and you're concerned about memory).
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54245618%2funexpected-behaviour-with-python-generator%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
7 Answers
7
active
oldest
votes
7 Answers
7
active
oldest
votes
active
oldest
votes
active
oldest
votes
As others have mentioned Python generators are lazy. When this line is run:
f = (x for x in array if array.count(x) == 2) # Filters original
nothing actually happens yet. You've just declared how the generator function f will work. Array is not looked at yet. Then, you create a new array that replaces the first one, and finally when you call
print(list(f)) # Outputs filtered
the generator now needs the actual values and starts pulling them from the generator f. But at this point, array already refers to the second one, so you get an empty list.
If you need to reassign the list, and can't use a different variable to hold it, consider creating the list instead of a generator in the second line:
f = [x for x in array if array.count(x) == 2] # Filters original
...
print(f)
add a comment |
As others have mentioned Python generators are lazy. When this line is run:
f = (x for x in array if array.count(x) == 2) # Filters original
nothing actually happens yet. You've just declared how the generator function f will work. Array is not looked at yet. Then, you create a new array that replaces the first one, and finally when you call
print(list(f)) # Outputs filtered
the generator now needs the actual values and starts pulling them from the generator f. But at this point, array already refers to the second one, so you get an empty list.
If you need to reassign the list, and can't use a different variable to hold it, consider creating the list instead of a generator in the second line:
f = [x for x in array if array.count(x) == 2] # Filters original
...
print(f)
add a comment |
As others have mentioned Python generators are lazy. When this line is run:
f = (x for x in array if array.count(x) == 2) # Filters original
nothing actually happens yet. You've just declared how the generator function f will work. Array is not looked at yet. Then, you create a new array that replaces the first one, and finally when you call
print(list(f)) # Outputs filtered
the generator now needs the actual values and starts pulling them from the generator f. But at this point, array already refers to the second one, so you get an empty list.
If you need to reassign the list, and can't use a different variable to hold it, consider creating the list instead of a generator in the second line:
f = [x for x in array if array.count(x) == 2] # Filters original
...
print(f)
As others have mentioned Python generators are lazy. When this line is run:
f = (x for x in array if array.count(x) == 2) # Filters original
nothing actually happens yet. You've just declared how the generator function f will work. Array is not looked at yet. Then, you create a new array that replaces the first one, and finally when you call
print(list(f)) # Outputs filtered
the generator now needs the actual values and starts pulling them from the generator f. But at this point, array already refers to the second one, so you get an empty list.
If you need to reassign the list, and can't use a different variable to hold it, consider creating the list instead of a generator in the second line:
f = [x for x in array if array.count(x) == 2] # Filters original
...
print(f)
edited 8 hours ago
answered 8 hours ago
StevenSteven
1067
1067
add a comment |
add a comment |
Generators are lazy, they won't be evaluated until you iterate through them. In this case that's at the point you create the list
with the generator as input, at the print
.
When am I iterating through them. Am I meant to?
– Suraj Kothari
8 hours ago
@SurajKothari when you create thelist
it will iterate for you without you needing to do it explicitly.
– Mark Ransom
8 hours ago
Also which list? When I declare the first one, or re-assign the second?
– Suraj Kothari
8 hours ago
What first & second? You define only one list, at the final line of your code.
– Prune
8 hours ago
@SurajKothari I updated the answer to be more clear.
– Mark Ransom
8 hours ago
add a comment |
Generators are lazy, they won't be evaluated until you iterate through them. In this case that's at the point you create the list
with the generator as input, at the print
.
When am I iterating through them. Am I meant to?
– Suraj Kothari
8 hours ago
@SurajKothari when you create thelist
it will iterate for you without you needing to do it explicitly.
– Mark Ransom
8 hours ago
Also which list? When I declare the first one, or re-assign the second?
– Suraj Kothari
8 hours ago
What first & second? You define only one list, at the final line of your code.
– Prune
8 hours ago
@SurajKothari I updated the answer to be more clear.
– Mark Ransom
8 hours ago
add a comment |
Generators are lazy, they won't be evaluated until you iterate through them. In this case that's at the point you create the list
with the generator as input, at the print
.
Generators are lazy, they won't be evaluated until you iterate through them. In this case that's at the point you create the list
with the generator as input, at the print
.
edited 8 hours ago
answered 8 hours ago
Mark RansomMark Ransom
223k29281508
223k29281508
When am I iterating through them. Am I meant to?
– Suraj Kothari
8 hours ago
@SurajKothari when you create thelist
it will iterate for you without you needing to do it explicitly.
– Mark Ransom
8 hours ago
Also which list? When I declare the first one, or re-assign the second?
– Suraj Kothari
8 hours ago
What first & second? You define only one list, at the final line of your code.
– Prune
8 hours ago
@SurajKothari I updated the answer to be more clear.
– Mark Ransom
8 hours ago
add a comment |
When am I iterating through them. Am I meant to?
– Suraj Kothari
8 hours ago
@SurajKothari when you create thelist
it will iterate for you without you needing to do it explicitly.
– Mark Ransom
8 hours ago
Also which list? When I declare the first one, or re-assign the second?
– Suraj Kothari
8 hours ago
What first & second? You define only one list, at the final line of your code.
– Prune
8 hours ago
@SurajKothari I updated the answer to be more clear.
– Mark Ransom
8 hours ago
When am I iterating through them. Am I meant to?
– Suraj Kothari
8 hours ago
When am I iterating through them. Am I meant to?
– Suraj Kothari
8 hours ago
@SurajKothari when you create the
list
it will iterate for you without you needing to do it explicitly.– Mark Ransom
8 hours ago
@SurajKothari when you create the
list
it will iterate for you without you needing to do it explicitly.– Mark Ransom
8 hours ago
Also which list? When I declare the first one, or re-assign the second?
– Suraj Kothari
8 hours ago
Also which list? When I declare the first one, or re-assign the second?
– Suraj Kothari
8 hours ago
What first & second? You define only one list, at the final line of your code.
– Prune
8 hours ago
What first & second? You define only one list, at the final line of your code.
– Prune
8 hours ago
@SurajKothari I updated the answer to be more clear.
– Mark Ransom
8 hours ago
@SurajKothari I updated the answer to be more clear.
– Mark Ransom
8 hours ago
add a comment |
You are not using a generator correctly if this is the primary use of this code. Use a list comprehension instead of a generator comprehension. Just replace the parentheses with brackets. It evaluates to a list if you don't know.
array = [1, 2, 2, 4, 5]
f = [x for x in array if array.count(x) == 2]
array = [5, 6, 1, 2, 9]
print(f)
#[2, 2]
You are getting this response because of the nature of a generator. You're calling the generator when it't contents will evaluate to
Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.
– Suraj Kothari
8 hours ago
With your change,list(f)
becomes redundant.
– Mark Ransom
8 hours ago
Lol @Mark Ransom, copy paste got me, I edited.
– Jaba
8 hours ago
@SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!
– Jaba
8 hours ago
add a comment |
You are not using a generator correctly if this is the primary use of this code. Use a list comprehension instead of a generator comprehension. Just replace the parentheses with brackets. It evaluates to a list if you don't know.
array = [1, 2, 2, 4, 5]
f = [x for x in array if array.count(x) == 2]
array = [5, 6, 1, 2, 9]
print(f)
#[2, 2]
You are getting this response because of the nature of a generator. You're calling the generator when it't contents will evaluate to
Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.
– Suraj Kothari
8 hours ago
With your change,list(f)
becomes redundant.
– Mark Ransom
8 hours ago
Lol @Mark Ransom, copy paste got me, I edited.
– Jaba
8 hours ago
@SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!
– Jaba
8 hours ago
add a comment |
You are not using a generator correctly if this is the primary use of this code. Use a list comprehension instead of a generator comprehension. Just replace the parentheses with brackets. It evaluates to a list if you don't know.
array = [1, 2, 2, 4, 5]
f = [x for x in array if array.count(x) == 2]
array = [5, 6, 1, 2, 9]
print(f)
#[2, 2]
You are getting this response because of the nature of a generator. You're calling the generator when it't contents will evaluate to
You are not using a generator correctly if this is the primary use of this code. Use a list comprehension instead of a generator comprehension. Just replace the parentheses with brackets. It evaluates to a list if you don't know.
array = [1, 2, 2, 4, 5]
f = [x for x in array if array.count(x) == 2]
array = [5, 6, 1, 2, 9]
print(f)
#[2, 2]
You are getting this response because of the nature of a generator. You're calling the generator when it't contents will evaluate to
edited 8 hours ago
answered 8 hours ago
JabaJaba
6,959175394
6,959175394
Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.
– Suraj Kothari
8 hours ago
With your change,list(f)
becomes redundant.
– Mark Ransom
8 hours ago
Lol @Mark Ransom, copy paste got me, I edited.
– Jaba
8 hours ago
@SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!
– Jaba
8 hours ago
add a comment |
Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.
– Suraj Kothari
8 hours ago
With your change,list(f)
becomes redundant.
– Mark Ransom
8 hours ago
Lol @Mark Ransom, copy paste got me, I edited.
– Jaba
8 hours ago
@SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!
– Jaba
8 hours ago
Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.
– Suraj Kothari
8 hours ago
Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.
– Suraj Kothari
8 hours ago
With your change,
list(f)
becomes redundant.– Mark Ransom
8 hours ago
With your change,
list(f)
becomes redundant.– Mark Ransom
8 hours ago
Lol @Mark Ransom, copy paste got me, I edited.
– Jaba
8 hours ago
Lol @Mark Ransom, copy paste got me, I edited.
– Jaba
8 hours ago
@SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!
– Jaba
8 hours ago
@SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!
– Jaba
8 hours ago
add a comment |
Generator evaluation is "lazy" -- it doesn't get executed until you actualize it with a proper reference. With your line:
Look again at your output with the type of f
: that object is a generator, not a sequence. It's waiting to be used, an iterator of sorts.
Your generator isn't evaluated until you start requiring values from it. At that point, it uses the available values at that point, not the point at which it was defined.
Code to "make it work"
That depends on what you mean by "make it work". If you want f
to be a filtered list, then use a list, not a generator:
f = [x for x in array if array.count(x) == 2] # Filters original
I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.
– Suraj Kothari
8 hours ago
add a comment |
Generator evaluation is "lazy" -- it doesn't get executed until you actualize it with a proper reference. With your line:
Look again at your output with the type of f
: that object is a generator, not a sequence. It's waiting to be used, an iterator of sorts.
Your generator isn't evaluated until you start requiring values from it. At that point, it uses the available values at that point, not the point at which it was defined.
Code to "make it work"
That depends on what you mean by "make it work". If you want f
to be a filtered list, then use a list, not a generator:
f = [x for x in array if array.count(x) == 2] # Filters original
I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.
– Suraj Kothari
8 hours ago
add a comment |
Generator evaluation is "lazy" -- it doesn't get executed until you actualize it with a proper reference. With your line:
Look again at your output with the type of f
: that object is a generator, not a sequence. It's waiting to be used, an iterator of sorts.
Your generator isn't evaluated until you start requiring values from it. At that point, it uses the available values at that point, not the point at which it was defined.
Code to "make it work"
That depends on what you mean by "make it work". If you want f
to be a filtered list, then use a list, not a generator:
f = [x for x in array if array.count(x) == 2] # Filters original
Generator evaluation is "lazy" -- it doesn't get executed until you actualize it with a proper reference. With your line:
Look again at your output with the type of f
: that object is a generator, not a sequence. It's waiting to be used, an iterator of sorts.
Your generator isn't evaluated until you start requiring values from it. At that point, it uses the available values at that point, not the point at which it was defined.
Code to "make it work"
That depends on what you mean by "make it work". If you want f
to be a filtered list, then use a list, not a generator:
f = [x for x in array if array.count(x) == 2] # Filters original
edited 7 hours ago
answered 8 hours ago
PrunePrune
43.2k143456
43.2k143456
I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.
– Suraj Kothari
8 hours ago
add a comment |
I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.
– Suraj Kothari
8 hours ago
I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.
– Suraj Kothari
8 hours ago
I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.
– Suraj Kothari
8 hours ago
add a comment |
Generators are lazy and your newly defined array
is used when you exhaust your generator after redefining. Therefore, the output is correct. A quick fix is to use a list comprehension by replacing parentheses ()
by brackets .
Moving on to how better to write your logic, counting a value in a loop has quadratic complexity. For an algorithm that works in linear time, you can use collections.Counter
to count values, and keep a copy of your original list:
from collections import Counter
array = [1, 2, 2, 4, 5] # original array
counts = Counter(array) # count each value in array
old_array = array.copy() # make copy
array = [5, 6, 1, 2, 9] # updates array
# order relevant
res = [x for x in old_array if counts[x] >= 2]
print(res)
# [2, 2]
# order irrelevant
from itertools import chain
res = list(chain.from_iterable([x]*count for x, count in counts.items() if count >= 2))
print(res)
# [2, 2]
Notice the second version doesn't even require old_array
and is useful if there is no need to maintain ordering of values in your original array.
add a comment |
Generators are lazy and your newly defined array
is used when you exhaust your generator after redefining. Therefore, the output is correct. A quick fix is to use a list comprehension by replacing parentheses ()
by brackets .
Moving on to how better to write your logic, counting a value in a loop has quadratic complexity. For an algorithm that works in linear time, you can use collections.Counter
to count values, and keep a copy of your original list:
from collections import Counter
array = [1, 2, 2, 4, 5] # original array
counts = Counter(array) # count each value in array
old_array = array.copy() # make copy
array = [5, 6, 1, 2, 9] # updates array
# order relevant
res = [x for x in old_array if counts[x] >= 2]
print(res)
# [2, 2]
# order irrelevant
from itertools import chain
res = list(chain.from_iterable([x]*count for x, count in counts.items() if count >= 2))
print(res)
# [2, 2]
Notice the second version doesn't even require old_array
and is useful if there is no need to maintain ordering of values in your original array.
add a comment |
Generators are lazy and your newly defined array
is used when you exhaust your generator after redefining. Therefore, the output is correct. A quick fix is to use a list comprehension by replacing parentheses ()
by brackets .
Moving on to how better to write your logic, counting a value in a loop has quadratic complexity. For an algorithm that works in linear time, you can use collections.Counter
to count values, and keep a copy of your original list:
from collections import Counter
array = [1, 2, 2, 4, 5] # original array
counts = Counter(array) # count each value in array
old_array = array.copy() # make copy
array = [5, 6, 1, 2, 9] # updates array
# order relevant
res = [x for x in old_array if counts[x] >= 2]
print(res)
# [2, 2]
# order irrelevant
from itertools import chain
res = list(chain.from_iterable([x]*count for x, count in counts.items() if count >= 2))
print(res)
# [2, 2]
Notice the second version doesn't even require old_array
and is useful if there is no need to maintain ordering of values in your original array.
Generators are lazy and your newly defined array
is used when you exhaust your generator after redefining. Therefore, the output is correct. A quick fix is to use a list comprehension by replacing parentheses ()
by brackets .
Moving on to how better to write your logic, counting a value in a loop has quadratic complexity. For an algorithm that works in linear time, you can use collections.Counter
to count values, and keep a copy of your original list:
from collections import Counter
array = [1, 2, 2, 4, 5] # original array
counts = Counter(array) # count each value in array
old_array = array.copy() # make copy
array = [5, 6, 1, 2, 9] # updates array
# order relevant
res = [x for x in old_array if counts[x] >= 2]
print(res)
# [2, 2]
# order irrelevant
from itertools import chain
res = list(chain.from_iterable([x]*count for x, count in counts.items() if count >= 2))
print(res)
# [2, 2]
Notice the second version doesn't even require old_array
and is useful if there is no need to maintain ordering of values in your original array.
answered 5 hours ago
jppjpp
95.7k2157109
95.7k2157109
add a comment |
add a comment |
The root cause of the problem is that generators are lazy; variables are evaluated each time:
>>> l = [1, 2, 2, 4, 5, 5, 5]
>>> filtered = (x for x in l if l.count(x) == 2)
>>> l = [1, 2, 4, 4, 5, 6, 6]
>>> list(filtered)
[4]
It iterates over the original list and evaluates the condition with the current list. In this case, 4 appeared twice in the new list, causing it to appear in the result. It only appears once in the result because it only appeared once in the original list. The 6s appear twice in the new list, but never appear in the old list and are hence never shown.
Full function introspection for the curious (the line with the comment is the important line):
>>> l = [1, 2, 2, 4, 5]
>>> filtered = (x for x in l if l.count(x) == 2)
>>> l = [1, 2, 4, 4, 5, 6, 6]
>>> list(filtered)
[4]
>>> def f(original, new, count):
current = original
filtered = (x for x in current if current.count(x) == count)
current = new
return list(filtered)
>>> from dis import dis
>>> dis(f)
2 0 LOAD_FAST 0 (original)
3 STORE_DEREF 1 (current)
3 6 LOAD_CLOSURE 0 (count)
9 LOAD_CLOSURE 1 (current)
12 BUILD_TUPLE 2
15 LOAD_CONST 1 (<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>)
18 LOAD_CONST 2 ('f.<locals>.<genexpr>')
21 MAKE_CLOSURE 0
24 LOAD_DEREF 1 (current)
27 GET_ITER
28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
31 STORE_FAST 3 (filtered)
4 34 LOAD_FAST 1 (new)
37 STORE_DEREF 1 (current)
5 40 LOAD_GLOBAL 0 (list)
43 LOAD_FAST 3 (filtered)
46 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
49 RETURN_VALUE
>>> f.__code__.co_varnames
('original', 'new', 'count', 'filtered')
>>> f.__code__.co_cellvars
('count', 'current')
>>> f.__code__.co_consts
(None, <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>, 'f.<locals>.<genexpr>')
>>> f.__code__.co_consts[1]
<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>
>>> dis(f.__code__.co_consts[1])
3 0 LOAD_FAST 0 (.0)
>> 3 FOR_ITER 32 (to 38)
6 STORE_FAST 1 (x)
9 LOAD_DEREF 1 (current) # This loads the current list every time, as opposed to loading a constant.
12 LOAD_ATTR 0 (count)
15 LOAD_FAST 1 (x)
18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
21 LOAD_DEREF 0 (count)
24 COMPARE_OP 2 (==)
27 POP_JUMP_IF_FALSE 3
30 LOAD_FAST 1 (x)
33 YIELD_VALUE
34 POP_TOP
35 JUMP_ABSOLUTE 3
>> 38 LOAD_CONST 0 (None)
41 RETURN_VALUE
>>> f.__code__.co_consts[1].co_consts
(None,)
To reiterate: The list to be iterated is only loaded once. Any closures in the condition or expression, however, are loaded from the enclosing scope each iteration. They are not stored in a constant.
The best solution for your problem would be to create a new variable referencing the original list and use that in your generator expression,.
add a comment |
The root cause of the problem is that generators are lazy; variables are evaluated each time:
>>> l = [1, 2, 2, 4, 5, 5, 5]
>>> filtered = (x for x in l if l.count(x) == 2)
>>> l = [1, 2, 4, 4, 5, 6, 6]
>>> list(filtered)
[4]
It iterates over the original list and evaluates the condition with the current list. In this case, 4 appeared twice in the new list, causing it to appear in the result. It only appears once in the result because it only appeared once in the original list. The 6s appear twice in the new list, but never appear in the old list and are hence never shown.
Full function introspection for the curious (the line with the comment is the important line):
>>> l = [1, 2, 2, 4, 5]
>>> filtered = (x for x in l if l.count(x) == 2)
>>> l = [1, 2, 4, 4, 5, 6, 6]
>>> list(filtered)
[4]
>>> def f(original, new, count):
current = original
filtered = (x for x in current if current.count(x) == count)
current = new
return list(filtered)
>>> from dis import dis
>>> dis(f)
2 0 LOAD_FAST 0 (original)
3 STORE_DEREF 1 (current)
3 6 LOAD_CLOSURE 0 (count)
9 LOAD_CLOSURE 1 (current)
12 BUILD_TUPLE 2
15 LOAD_CONST 1 (<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>)
18 LOAD_CONST 2 ('f.<locals>.<genexpr>')
21 MAKE_CLOSURE 0
24 LOAD_DEREF 1 (current)
27 GET_ITER
28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
31 STORE_FAST 3 (filtered)
4 34 LOAD_FAST 1 (new)
37 STORE_DEREF 1 (current)
5 40 LOAD_GLOBAL 0 (list)
43 LOAD_FAST 3 (filtered)
46 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
49 RETURN_VALUE
>>> f.__code__.co_varnames
('original', 'new', 'count', 'filtered')
>>> f.__code__.co_cellvars
('count', 'current')
>>> f.__code__.co_consts
(None, <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>, 'f.<locals>.<genexpr>')
>>> f.__code__.co_consts[1]
<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>
>>> dis(f.__code__.co_consts[1])
3 0 LOAD_FAST 0 (.0)
>> 3 FOR_ITER 32 (to 38)
6 STORE_FAST 1 (x)
9 LOAD_DEREF 1 (current) # This loads the current list every time, as opposed to loading a constant.
12 LOAD_ATTR 0 (count)
15 LOAD_FAST 1 (x)
18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
21 LOAD_DEREF 0 (count)
24 COMPARE_OP 2 (==)
27 POP_JUMP_IF_FALSE 3
30 LOAD_FAST 1 (x)
33 YIELD_VALUE
34 POP_TOP
35 JUMP_ABSOLUTE 3
>> 38 LOAD_CONST 0 (None)
41 RETURN_VALUE
>>> f.__code__.co_consts[1].co_consts
(None,)
To reiterate: The list to be iterated is only loaded once. Any closures in the condition or expression, however, are loaded from the enclosing scope each iteration. They are not stored in a constant.
The best solution for your problem would be to create a new variable referencing the original list and use that in your generator expression,.
add a comment |
The root cause of the problem is that generators are lazy; variables are evaluated each time:
>>> l = [1, 2, 2, 4, 5, 5, 5]
>>> filtered = (x for x in l if l.count(x) == 2)
>>> l = [1, 2, 4, 4, 5, 6, 6]
>>> list(filtered)
[4]
It iterates over the original list and evaluates the condition with the current list. In this case, 4 appeared twice in the new list, causing it to appear in the result. It only appears once in the result because it only appeared once in the original list. The 6s appear twice in the new list, but never appear in the old list and are hence never shown.
Full function introspection for the curious (the line with the comment is the important line):
>>> l = [1, 2, 2, 4, 5]
>>> filtered = (x for x in l if l.count(x) == 2)
>>> l = [1, 2, 4, 4, 5, 6, 6]
>>> list(filtered)
[4]
>>> def f(original, new, count):
current = original
filtered = (x for x in current if current.count(x) == count)
current = new
return list(filtered)
>>> from dis import dis
>>> dis(f)
2 0 LOAD_FAST 0 (original)
3 STORE_DEREF 1 (current)
3 6 LOAD_CLOSURE 0 (count)
9 LOAD_CLOSURE 1 (current)
12 BUILD_TUPLE 2
15 LOAD_CONST 1 (<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>)
18 LOAD_CONST 2 ('f.<locals>.<genexpr>')
21 MAKE_CLOSURE 0
24 LOAD_DEREF 1 (current)
27 GET_ITER
28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
31 STORE_FAST 3 (filtered)
4 34 LOAD_FAST 1 (new)
37 STORE_DEREF 1 (current)
5 40 LOAD_GLOBAL 0 (list)
43 LOAD_FAST 3 (filtered)
46 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
49 RETURN_VALUE
>>> f.__code__.co_varnames
('original', 'new', 'count', 'filtered')
>>> f.__code__.co_cellvars
('count', 'current')
>>> f.__code__.co_consts
(None, <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>, 'f.<locals>.<genexpr>')
>>> f.__code__.co_consts[1]
<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>
>>> dis(f.__code__.co_consts[1])
3 0 LOAD_FAST 0 (.0)
>> 3 FOR_ITER 32 (to 38)
6 STORE_FAST 1 (x)
9 LOAD_DEREF 1 (current) # This loads the current list every time, as opposed to loading a constant.
12 LOAD_ATTR 0 (count)
15 LOAD_FAST 1 (x)
18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
21 LOAD_DEREF 0 (count)
24 COMPARE_OP 2 (==)
27 POP_JUMP_IF_FALSE 3
30 LOAD_FAST 1 (x)
33 YIELD_VALUE
34 POP_TOP
35 JUMP_ABSOLUTE 3
>> 38 LOAD_CONST 0 (None)
41 RETURN_VALUE
>>> f.__code__.co_consts[1].co_consts
(None,)
To reiterate: The list to be iterated is only loaded once. Any closures in the condition or expression, however, are loaded from the enclosing scope each iteration. They are not stored in a constant.
The best solution for your problem would be to create a new variable referencing the original list and use that in your generator expression,.
The root cause of the problem is that generators are lazy; variables are evaluated each time:
>>> l = [1, 2, 2, 4, 5, 5, 5]
>>> filtered = (x for x in l if l.count(x) == 2)
>>> l = [1, 2, 4, 4, 5, 6, 6]
>>> list(filtered)
[4]
It iterates over the original list and evaluates the condition with the current list. In this case, 4 appeared twice in the new list, causing it to appear in the result. It only appears once in the result because it only appeared once in the original list. The 6s appear twice in the new list, but never appear in the old list and are hence never shown.
Full function introspection for the curious (the line with the comment is the important line):
>>> l = [1, 2, 2, 4, 5]
>>> filtered = (x for x in l if l.count(x) == 2)
>>> l = [1, 2, 4, 4, 5, 6, 6]
>>> list(filtered)
[4]
>>> def f(original, new, count):
current = original
filtered = (x for x in current if current.count(x) == count)
current = new
return list(filtered)
>>> from dis import dis
>>> dis(f)
2 0 LOAD_FAST 0 (original)
3 STORE_DEREF 1 (current)
3 6 LOAD_CLOSURE 0 (count)
9 LOAD_CLOSURE 1 (current)
12 BUILD_TUPLE 2
15 LOAD_CONST 1 (<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>)
18 LOAD_CONST 2 ('f.<locals>.<genexpr>')
21 MAKE_CLOSURE 0
24 LOAD_DEREF 1 (current)
27 GET_ITER
28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
31 STORE_FAST 3 (filtered)
4 34 LOAD_FAST 1 (new)
37 STORE_DEREF 1 (current)
5 40 LOAD_GLOBAL 0 (list)
43 LOAD_FAST 3 (filtered)
46 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
49 RETURN_VALUE
>>> f.__code__.co_varnames
('original', 'new', 'count', 'filtered')
>>> f.__code__.co_cellvars
('count', 'current')
>>> f.__code__.co_consts
(None, <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>, 'f.<locals>.<genexpr>')
>>> f.__code__.co_consts[1]
<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>
>>> dis(f.__code__.co_consts[1])
3 0 LOAD_FAST 0 (.0)
>> 3 FOR_ITER 32 (to 38)
6 STORE_FAST 1 (x)
9 LOAD_DEREF 1 (current) # This loads the current list every time, as opposed to loading a constant.
12 LOAD_ATTR 0 (count)
15 LOAD_FAST 1 (x)
18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
21 LOAD_DEREF 0 (count)
24 COMPARE_OP 2 (==)
27 POP_JUMP_IF_FALSE 3
30 LOAD_FAST 1 (x)
33 YIELD_VALUE
34 POP_TOP
35 JUMP_ABSOLUTE 3
>> 38 LOAD_CONST 0 (None)
41 RETURN_VALUE
>>> f.__code__.co_consts[1].co_consts
(None,)
To reiterate: The list to be iterated is only loaded once. Any closures in the condition or expression, however, are loaded from the enclosing scope each iteration. They are not stored in a constant.
The best solution for your problem would be to create a new variable referencing the original list and use that in your generator expression,.
edited 3 hours ago
answered 4 hours ago
Solomon UckoSolomon Ucko
6271719
6271719
add a comment |
add a comment |
Others have already explained the root cause of the issue - the generator is binding to the name of the array
local variable, rather than its value.
The most pythonic solution is definitely the list comprehension:
f = [x for x in array if array.count(x) == 2]
However, if there is some reason that you don't want to create a list, you can also force a scope close over array
:
f = (lambda array=array: (x for x in array if array.count(x) == 2))()
What's happening here is that the lambda
captures the reference to array
at the time the line is run, ensuring that the generator sees the variable you expect, even if the variable is later redefined.
Note that this still binds to the variable (reference), not the value, so, for example, the following will print [2, 2, 4, 4]
:
array = [1, 2, 2, 4, 5] # Original array
f = (lambda array=array: (x for x in array if array.count(x) == 2))() # Close over array
array.append(4) # This *will* be captured
array = [5, 6, 1, 2, 9] # Updates original to something else
print(list(f)) # Outputs [2, 2, 4, 4]
This is a common pattern in some languages, but it's not very pythonic, so only really makes sense if there's a very good reason for not using the list comprehension (e.g., if array
is very long, or is being used in a nested generator comprehension, and you're concerned about memory).
add a comment |
Others have already explained the root cause of the issue - the generator is binding to the name of the array
local variable, rather than its value.
The most pythonic solution is definitely the list comprehension:
f = [x for x in array if array.count(x) == 2]
However, if there is some reason that you don't want to create a list, you can also force a scope close over array
:
f = (lambda array=array: (x for x in array if array.count(x) == 2))()
What's happening here is that the lambda
captures the reference to array
at the time the line is run, ensuring that the generator sees the variable you expect, even if the variable is later redefined.
Note that this still binds to the variable (reference), not the value, so, for example, the following will print [2, 2, 4, 4]
:
array = [1, 2, 2, 4, 5] # Original array
f = (lambda array=array: (x for x in array if array.count(x) == 2))() # Close over array
array.append(4) # This *will* be captured
array = [5, 6, 1, 2, 9] # Updates original to something else
print(list(f)) # Outputs [2, 2, 4, 4]
This is a common pattern in some languages, but it's not very pythonic, so only really makes sense if there's a very good reason for not using the list comprehension (e.g., if array
is very long, or is being used in a nested generator comprehension, and you're concerned about memory).
add a comment |
Others have already explained the root cause of the issue - the generator is binding to the name of the array
local variable, rather than its value.
The most pythonic solution is definitely the list comprehension:
f = [x for x in array if array.count(x) == 2]
However, if there is some reason that you don't want to create a list, you can also force a scope close over array
:
f = (lambda array=array: (x for x in array if array.count(x) == 2))()
What's happening here is that the lambda
captures the reference to array
at the time the line is run, ensuring that the generator sees the variable you expect, even if the variable is later redefined.
Note that this still binds to the variable (reference), not the value, so, for example, the following will print [2, 2, 4, 4]
:
array = [1, 2, 2, 4, 5] # Original array
f = (lambda array=array: (x for x in array if array.count(x) == 2))() # Close over array
array.append(4) # This *will* be captured
array = [5, 6, 1, 2, 9] # Updates original to something else
print(list(f)) # Outputs [2, 2, 4, 4]
This is a common pattern in some languages, but it's not very pythonic, so only really makes sense if there's a very good reason for not using the list comprehension (e.g., if array
is very long, or is being used in a nested generator comprehension, and you're concerned about memory).
Others have already explained the root cause of the issue - the generator is binding to the name of the array
local variable, rather than its value.
The most pythonic solution is definitely the list comprehension:
f = [x for x in array if array.count(x) == 2]
However, if there is some reason that you don't want to create a list, you can also force a scope close over array
:
f = (lambda array=array: (x for x in array if array.count(x) == 2))()
What's happening here is that the lambda
captures the reference to array
at the time the line is run, ensuring that the generator sees the variable you expect, even if the variable is later redefined.
Note that this still binds to the variable (reference), not the value, so, for example, the following will print [2, 2, 4, 4]
:
array = [1, 2, 2, 4, 5] # Original array
f = (lambda array=array: (x for x in array if array.count(x) == 2))() # Close over array
array.append(4) # This *will* be captured
array = [5, 6, 1, 2, 9] # Updates original to something else
print(list(f)) # Outputs [2, 2, 4, 4]
This is a common pattern in some languages, but it's not very pythonic, so only really makes sense if there's a very good reason for not using the list comprehension (e.g., if array
is very long, or is being used in a nested generator comprehension, and you're concerned about memory).
answered 17 mins ago
sapisapi
6,57963161
6,57963161
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54245618%2funexpected-behaviour-with-python-generator%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
You keep printing
list(g)
, butg
is an undefined symbol. Perhaps you have a typo, usingg
forf
?– Prune
8 hours ago
Yh that is what I meant sorry
– Suraj Kothari
8 hours ago
3
You redefine
array
and your new array is what gets referenced by the lazy generator comprehension.– jpp
8 hours ago
Would be good to see an answer that mentions scope.
– trailing_whitespace
1 hour ago
1
Slap "strange" onto the title and you're sure to get a few upvotes.
– coldspeed
1 hour ago