Unexpected behaviour with Python generator












13















I was running a piece of code that unexpectedly gave a logic error at one part of the program. When investigating the section, I created a test file to test the set of statements being run and found out an unusual bug that seems very odd.



I tested this simple code:



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original to something else

print(list(f)) # Outputs filtered


And the output was:



>>> 


Yes, nothing. I was expecting the filter comprehension to get items in the array with a count of 2 and output this, but I didn't get that:



# Expected output
>>> [2, 2]


When I commented out the third line to test it once again:



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
### array = [5, 6, 1, 2, 9] # Ignore line

print(list(f)) # Outputs filtered


The output was correct (you can test it for yourself):



>>> [2, 2]


At one point I outputted the type of the variable 'f':



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original

print(type(f))
print(list(f)) # Outputs filtered


And I got:



>>> <class 'generator'>
>>>


TL;DR: Why is updating a list in Python changing the output of another generator variable? This seems very odd to me.










share|improve this question




















  • 2





    You keep printing list(g), but g is an undefined symbol. Perhaps you have a typo, using g for f?

    – Prune
    8 hours ago











  • Yh that is what I meant sorry

    – Suraj Kothari
    8 hours ago






  • 3





    You redefine array and your new array is what gets referenced by the lazy generator comprehension.

    – jpp
    8 hours ago













  • Would be good to see an answer that mentions scope.

    – trailing_whitespace
    1 hour ago






  • 1





    Slap "strange" onto the title and you're sure to get a few upvotes.

    – coldspeed
    1 hour ago


















13















I was running a piece of code that unexpectedly gave a logic error at one part of the program. When investigating the section, I created a test file to test the set of statements being run and found out an unusual bug that seems very odd.



I tested this simple code:



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original to something else

print(list(f)) # Outputs filtered


And the output was:



>>> 


Yes, nothing. I was expecting the filter comprehension to get items in the array with a count of 2 and output this, but I didn't get that:



# Expected output
>>> [2, 2]


When I commented out the third line to test it once again:



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
### array = [5, 6, 1, 2, 9] # Ignore line

print(list(f)) # Outputs filtered


The output was correct (you can test it for yourself):



>>> [2, 2]


At one point I outputted the type of the variable 'f':



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original

print(type(f))
print(list(f)) # Outputs filtered


And I got:



>>> <class 'generator'>
>>>


TL;DR: Why is updating a list in Python changing the output of another generator variable? This seems very odd to me.










share|improve this question




















  • 2





    You keep printing list(g), but g is an undefined symbol. Perhaps you have a typo, using g for f?

    – Prune
    8 hours ago











  • Yh that is what I meant sorry

    – Suraj Kothari
    8 hours ago






  • 3





    You redefine array and your new array is what gets referenced by the lazy generator comprehension.

    – jpp
    8 hours ago













  • Would be good to see an answer that mentions scope.

    – trailing_whitespace
    1 hour ago






  • 1





    Slap "strange" onto the title and you're sure to get a few upvotes.

    – coldspeed
    1 hour ago
















13












13








13


1






I was running a piece of code that unexpectedly gave a logic error at one part of the program. When investigating the section, I created a test file to test the set of statements being run and found out an unusual bug that seems very odd.



I tested this simple code:



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original to something else

print(list(f)) # Outputs filtered


And the output was:



>>> 


Yes, nothing. I was expecting the filter comprehension to get items in the array with a count of 2 and output this, but I didn't get that:



# Expected output
>>> [2, 2]


When I commented out the third line to test it once again:



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
### array = [5, 6, 1, 2, 9] # Ignore line

print(list(f)) # Outputs filtered


The output was correct (you can test it for yourself):



>>> [2, 2]


At one point I outputted the type of the variable 'f':



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original

print(type(f))
print(list(f)) # Outputs filtered


And I got:



>>> <class 'generator'>
>>>


TL;DR: Why is updating a list in Python changing the output of another generator variable? This seems very odd to me.










share|improve this question
















I was running a piece of code that unexpectedly gave a logic error at one part of the program. When investigating the section, I created a test file to test the set of statements being run and found out an unusual bug that seems very odd.



I tested this simple code:



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original to something else

print(list(f)) # Outputs filtered


And the output was:



>>> 


Yes, nothing. I was expecting the filter comprehension to get items in the array with a count of 2 and output this, but I didn't get that:



# Expected output
>>> [2, 2]


When I commented out the third line to test it once again:



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
### array = [5, 6, 1, 2, 9] # Ignore line

print(list(f)) # Outputs filtered


The output was correct (you can test it for yourself):



>>> [2, 2]


At one point I outputted the type of the variable 'f':



array = [1, 2, 2, 4, 5] # Original array
f = (x for x in array if array.count(x) == 2) # Filters original
array = [5, 6, 1, 2, 9] # Updates original

print(type(f))
print(list(f)) # Outputs filtered


And I got:



>>> <class 'generator'>
>>>


TL;DR: Why is updating a list in Python changing the output of another generator variable? This seems very odd to me.







python






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 48 mins ago









Ian Kemp

16.5k126798




16.5k126798










asked 8 hours ago









Suraj KothariSuraj Kothari

471213




471213








  • 2





    You keep printing list(g), but g is an undefined symbol. Perhaps you have a typo, using g for f?

    – Prune
    8 hours ago











  • Yh that is what I meant sorry

    – Suraj Kothari
    8 hours ago






  • 3





    You redefine array and your new array is what gets referenced by the lazy generator comprehension.

    – jpp
    8 hours ago













  • Would be good to see an answer that mentions scope.

    – trailing_whitespace
    1 hour ago






  • 1





    Slap "strange" onto the title and you're sure to get a few upvotes.

    – coldspeed
    1 hour ago
















  • 2





    You keep printing list(g), but g is an undefined symbol. Perhaps you have a typo, using g for f?

    – Prune
    8 hours ago











  • Yh that is what I meant sorry

    – Suraj Kothari
    8 hours ago






  • 3





    You redefine array and your new array is what gets referenced by the lazy generator comprehension.

    – jpp
    8 hours ago













  • Would be good to see an answer that mentions scope.

    – trailing_whitespace
    1 hour ago






  • 1





    Slap "strange" onto the title and you're sure to get a few upvotes.

    – coldspeed
    1 hour ago










2




2





You keep printing list(g), but g is an undefined symbol. Perhaps you have a typo, using g for f?

– Prune
8 hours ago





You keep printing list(g), but g is an undefined symbol. Perhaps you have a typo, using g for f?

– Prune
8 hours ago













Yh that is what I meant sorry

– Suraj Kothari
8 hours ago





Yh that is what I meant sorry

– Suraj Kothari
8 hours ago




3




3





You redefine array and your new array is what gets referenced by the lazy generator comprehension.

– jpp
8 hours ago







You redefine array and your new array is what gets referenced by the lazy generator comprehension.

– jpp
8 hours ago















Would be good to see an answer that mentions scope.

– trailing_whitespace
1 hour ago





Would be good to see an answer that mentions scope.

– trailing_whitespace
1 hour ago




1




1





Slap "strange" onto the title and you're sure to get a few upvotes.

– coldspeed
1 hour ago







Slap "strange" onto the title and you're sure to get a few upvotes.

– coldspeed
1 hour ago














7 Answers
7






active

oldest

votes


















9














As others have mentioned Python generators are lazy. When this line is run:



f = (x for x in array if array.count(x) == 2) # Filters original


nothing actually happens yet. You've just declared how the generator function f will work. Array is not looked at yet. Then, you create a new array that replaces the first one, and finally when you call



print(list(f)) # Outputs filtered


the generator now needs the actual values and starts pulling them from the generator f. But at this point, array already refers to the second one, so you get an empty list.



If you need to reassign the list, and can't use a different variable to hold it, consider creating the list instead of a generator in the second line:



f = [x for x in array if array.count(x) == 2] # Filters original
...
print(f)





share|improve this answer

































    3














    Generators are lazy, they won't be evaluated until you iterate through them. In this case that's at the point you create the list with the generator as input, at the print.






    share|improve this answer


























    • When am I iterating through them. Am I meant to?

      – Suraj Kothari
      8 hours ago











    • @SurajKothari when you create the list it will iterate for you without you needing to do it explicitly.

      – Mark Ransom
      8 hours ago











    • Also which list? When I declare the first one, or re-assign the second?

      – Suraj Kothari
      8 hours ago











    • What first & second? You define only one list, at the final line of your code.

      – Prune
      8 hours ago











    • @SurajKothari I updated the answer to be more clear.

      – Mark Ransom
      8 hours ago



















    2














    You are not using a generator correctly if this is the primary use of this code. Use a list comprehension instead of a generator comprehension. Just replace the parentheses with brackets. It evaluates to a list if you don't know.



    array = [1, 2, 2, 4, 5]
    f = [x for x in array if array.count(x) == 2]
    array = [5, 6, 1, 2, 9]

    print(f)
    #[2, 2]


    You are getting this response because of the nature of a generator. You're calling the generator when it't contents will evaluate to






    share|improve this answer


























    • Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.

      – Suraj Kothari
      8 hours ago











    • With your change, list(f) becomes redundant.

      – Mark Ransom
      8 hours ago











    • Lol @Mark Ransom, copy paste got me, I edited.

      – Jaba
      8 hours ago













    • @SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!

      – Jaba
      8 hours ago



















    0














    Generator evaluation is "lazy" -- it doesn't get executed until you actualize it with a proper reference. With your line:



    Look again at your output with the type of f: that object is a generator, not a sequence. It's waiting to be used, an iterator of sorts.



    Your generator isn't evaluated until you start requiring values from it. At that point, it uses the available values at that point, not the point at which it was defined.





    Code to "make it work"



    That depends on what you mean by "make it work". If you want f to be a filtered list, then use a list, not a generator:



    f = [x for x in array if array.count(x) == 2] # Filters original





    share|improve this answer


























    • I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.

      – Suraj Kothari
      8 hours ago



















    0














    Generators are lazy and your newly defined array is used when you exhaust your generator after redefining. Therefore, the output is correct. A quick fix is to use a list comprehension by replacing parentheses () by brackets .



    Moving on to how better to write your logic, counting a value in a loop has quadratic complexity. For an algorithm that works in linear time, you can use collections.Counter to count values, and keep a copy of your original list:



    from collections import Counter

    array = [1, 2, 2, 4, 5] # original array
    counts = Counter(array) # count each value in array
    old_array = array.copy() # make copy
    array = [5, 6, 1, 2, 9] # updates array

    # order relevant
    res = [x for x in old_array if counts[x] >= 2]
    print(res)
    # [2, 2]

    # order irrelevant
    from itertools import chain
    res = list(chain.from_iterable([x]*count for x, count in counts.items() if count >= 2))
    print(res)
    # [2, 2]


    Notice the second version doesn't even require old_array and is useful if there is no need to maintain ordering of values in your original array.






    share|improve this answer































      0














      The root cause of the problem is that generators are lazy; variables are evaluated each time:



      >>> l = [1, 2, 2, 4, 5, 5, 5]
      >>> filtered = (x for x in l if l.count(x) == 2)
      >>> l = [1, 2, 4, 4, 5, 6, 6]
      >>> list(filtered)
      [4]


      It iterates over the original list and evaluates the condition with the current list. In this case, 4 appeared twice in the new list, causing it to appear in the result. It only appears once in the result because it only appeared once in the original list. The 6s appear twice in the new list, but never appear in the old list and are hence never shown.



      Full function introspection for the curious (the line with the comment is the important line):



      >>> l = [1, 2, 2, 4, 5]
      >>> filtered = (x for x in l if l.count(x) == 2)
      >>> l = [1, 2, 4, 4, 5, 6, 6]
      >>> list(filtered)
      [4]
      >>> def f(original, new, count):
      current = original
      filtered = (x for x in current if current.count(x) == count)
      current = new
      return list(filtered)

      >>> from dis import dis
      >>> dis(f)
      2 0 LOAD_FAST 0 (original)
      3 STORE_DEREF 1 (current)

      3 6 LOAD_CLOSURE 0 (count)
      9 LOAD_CLOSURE 1 (current)
      12 BUILD_TUPLE 2
      15 LOAD_CONST 1 (<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>)
      18 LOAD_CONST 2 ('f.<locals>.<genexpr>')
      21 MAKE_CLOSURE 0
      24 LOAD_DEREF 1 (current)
      27 GET_ITER
      28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
      31 STORE_FAST 3 (filtered)

      4 34 LOAD_FAST 1 (new)
      37 STORE_DEREF 1 (current)

      5 40 LOAD_GLOBAL 0 (list)
      43 LOAD_FAST 3 (filtered)
      46 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
      49 RETURN_VALUE
      >>> f.__code__.co_varnames
      ('original', 'new', 'count', 'filtered')
      >>> f.__code__.co_cellvars
      ('count', 'current')
      >>> f.__code__.co_consts
      (None, <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>, 'f.<locals>.<genexpr>')
      >>> f.__code__.co_consts[1]
      <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>
      >>> dis(f.__code__.co_consts[1])
      3 0 LOAD_FAST 0 (.0)
      >> 3 FOR_ITER 32 (to 38)
      6 STORE_FAST 1 (x)
      9 LOAD_DEREF 1 (current) # This loads the current list every time, as opposed to loading a constant.
      12 LOAD_ATTR 0 (count)
      15 LOAD_FAST 1 (x)
      18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
      21 LOAD_DEREF 0 (count)
      24 COMPARE_OP 2 (==)
      27 POP_JUMP_IF_FALSE 3
      30 LOAD_FAST 1 (x)
      33 YIELD_VALUE
      34 POP_TOP
      35 JUMP_ABSOLUTE 3
      >> 38 LOAD_CONST 0 (None)
      41 RETURN_VALUE
      >>> f.__code__.co_consts[1].co_consts
      (None,)


      To reiterate: The list to be iterated is only loaded once. Any closures in the condition or expression, however, are loaded from the enclosing scope each iteration. They are not stored in a constant.



      The best solution for your problem would be to create a new variable referencing the original list and use that in your generator expression,.






      share|improve this answer

































        0














        Others have already explained the root cause of the issue - the generator is binding to the name of the array local variable, rather than its value.



        The most pythonic solution is definitely the list comprehension:



        f = [x for x in array if array.count(x) == 2]




        However, if there is some reason that you don't want to create a list, you can also force a scope close over array:



        f = (lambda array=array: (x for x in array if array.count(x) == 2))()


        What's happening here is that the lambda captures the reference to array at the time the line is run, ensuring that the generator sees the variable you expect, even if the variable is later redefined.



        Note that this still binds to the variable (reference), not the value, so, for example, the following will print [2, 2, 4, 4]:



        array = [1, 2, 2, 4, 5] # Original array

        f = (lambda array=array: (x for x in array if array.count(x) == 2))() # Close over array
        array.append(4) # This *will* be captured

        array = [5, 6, 1, 2, 9] # Updates original to something else

        print(list(f)) # Outputs [2, 2, 4, 4]




        This is a common pattern in some languages, but it's not very pythonic, so only really makes sense if there's a very good reason for not using the list comprehension (e.g., if array is very long, or is being used in a nested generator comprehension, and you're concerned about memory).






        share|improve this answer























          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54245618%2funexpected-behaviour-with-python-generator%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          7 Answers
          7






          active

          oldest

          votes








          7 Answers
          7






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          9














          As others have mentioned Python generators are lazy. When this line is run:



          f = (x for x in array if array.count(x) == 2) # Filters original


          nothing actually happens yet. You've just declared how the generator function f will work. Array is not looked at yet. Then, you create a new array that replaces the first one, and finally when you call



          print(list(f)) # Outputs filtered


          the generator now needs the actual values and starts pulling them from the generator f. But at this point, array already refers to the second one, so you get an empty list.



          If you need to reassign the list, and can't use a different variable to hold it, consider creating the list instead of a generator in the second line:



          f = [x for x in array if array.count(x) == 2] # Filters original
          ...
          print(f)





          share|improve this answer






























            9














            As others have mentioned Python generators are lazy. When this line is run:



            f = (x for x in array if array.count(x) == 2) # Filters original


            nothing actually happens yet. You've just declared how the generator function f will work. Array is not looked at yet. Then, you create a new array that replaces the first one, and finally when you call



            print(list(f)) # Outputs filtered


            the generator now needs the actual values and starts pulling them from the generator f. But at this point, array already refers to the second one, so you get an empty list.



            If you need to reassign the list, and can't use a different variable to hold it, consider creating the list instead of a generator in the second line:



            f = [x for x in array if array.count(x) == 2] # Filters original
            ...
            print(f)





            share|improve this answer




























              9












              9








              9







              As others have mentioned Python generators are lazy. When this line is run:



              f = (x for x in array if array.count(x) == 2) # Filters original


              nothing actually happens yet. You've just declared how the generator function f will work. Array is not looked at yet. Then, you create a new array that replaces the first one, and finally when you call



              print(list(f)) # Outputs filtered


              the generator now needs the actual values and starts pulling them from the generator f. But at this point, array already refers to the second one, so you get an empty list.



              If you need to reassign the list, and can't use a different variable to hold it, consider creating the list instead of a generator in the second line:



              f = [x for x in array if array.count(x) == 2] # Filters original
              ...
              print(f)





              share|improve this answer















              As others have mentioned Python generators are lazy. When this line is run:



              f = (x for x in array if array.count(x) == 2) # Filters original


              nothing actually happens yet. You've just declared how the generator function f will work. Array is not looked at yet. Then, you create a new array that replaces the first one, and finally when you call



              print(list(f)) # Outputs filtered


              the generator now needs the actual values and starts pulling them from the generator f. But at this point, array already refers to the second one, so you get an empty list.



              If you need to reassign the list, and can't use a different variable to hold it, consider creating the list instead of a generator in the second line:



              f = [x for x in array if array.count(x) == 2] # Filters original
              ...
              print(f)






              share|improve this answer














              share|improve this answer



              share|improve this answer








              edited 8 hours ago

























              answered 8 hours ago









              StevenSteven

              1067




              1067

























                  3














                  Generators are lazy, they won't be evaluated until you iterate through them. In this case that's at the point you create the list with the generator as input, at the print.






                  share|improve this answer


























                  • When am I iterating through them. Am I meant to?

                    – Suraj Kothari
                    8 hours ago











                  • @SurajKothari when you create the list it will iterate for you without you needing to do it explicitly.

                    – Mark Ransom
                    8 hours ago











                  • Also which list? When I declare the first one, or re-assign the second?

                    – Suraj Kothari
                    8 hours ago











                  • What first & second? You define only one list, at the final line of your code.

                    – Prune
                    8 hours ago











                  • @SurajKothari I updated the answer to be more clear.

                    – Mark Ransom
                    8 hours ago
















                  3














                  Generators are lazy, they won't be evaluated until you iterate through them. In this case that's at the point you create the list with the generator as input, at the print.






                  share|improve this answer


























                  • When am I iterating through them. Am I meant to?

                    – Suraj Kothari
                    8 hours ago











                  • @SurajKothari when you create the list it will iterate for you without you needing to do it explicitly.

                    – Mark Ransom
                    8 hours ago











                  • Also which list? When I declare the first one, or re-assign the second?

                    – Suraj Kothari
                    8 hours ago











                  • What first & second? You define only one list, at the final line of your code.

                    – Prune
                    8 hours ago











                  • @SurajKothari I updated the answer to be more clear.

                    – Mark Ransom
                    8 hours ago














                  3












                  3








                  3







                  Generators are lazy, they won't be evaluated until you iterate through them. In this case that's at the point you create the list with the generator as input, at the print.






                  share|improve this answer















                  Generators are lazy, they won't be evaluated until you iterate through them. In this case that's at the point you create the list with the generator as input, at the print.







                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited 8 hours ago

























                  answered 8 hours ago









                  Mark RansomMark Ransom

                  223k29281508




                  223k29281508













                  • When am I iterating through them. Am I meant to?

                    – Suraj Kothari
                    8 hours ago











                  • @SurajKothari when you create the list it will iterate for you without you needing to do it explicitly.

                    – Mark Ransom
                    8 hours ago











                  • Also which list? When I declare the first one, or re-assign the second?

                    – Suraj Kothari
                    8 hours ago











                  • What first & second? You define only one list, at the final line of your code.

                    – Prune
                    8 hours ago











                  • @SurajKothari I updated the answer to be more clear.

                    – Mark Ransom
                    8 hours ago



















                  • When am I iterating through them. Am I meant to?

                    – Suraj Kothari
                    8 hours ago











                  • @SurajKothari when you create the list it will iterate for you without you needing to do it explicitly.

                    – Mark Ransom
                    8 hours ago











                  • Also which list? When I declare the first one, or re-assign the second?

                    – Suraj Kothari
                    8 hours ago











                  • What first & second? You define only one list, at the final line of your code.

                    – Prune
                    8 hours ago











                  • @SurajKothari I updated the answer to be more clear.

                    – Mark Ransom
                    8 hours ago

















                  When am I iterating through them. Am I meant to?

                  – Suraj Kothari
                  8 hours ago





                  When am I iterating through them. Am I meant to?

                  – Suraj Kothari
                  8 hours ago













                  @SurajKothari when you create the list it will iterate for you without you needing to do it explicitly.

                  – Mark Ransom
                  8 hours ago





                  @SurajKothari when you create the list it will iterate for you without you needing to do it explicitly.

                  – Mark Ransom
                  8 hours ago













                  Also which list? When I declare the first one, or re-assign the second?

                  – Suraj Kothari
                  8 hours ago





                  Also which list? When I declare the first one, or re-assign the second?

                  – Suraj Kothari
                  8 hours ago













                  What first & second? You define only one list, at the final line of your code.

                  – Prune
                  8 hours ago





                  What first & second? You define only one list, at the final line of your code.

                  – Prune
                  8 hours ago













                  @SurajKothari I updated the answer to be more clear.

                  – Mark Ransom
                  8 hours ago





                  @SurajKothari I updated the answer to be more clear.

                  – Mark Ransom
                  8 hours ago











                  2














                  You are not using a generator correctly if this is the primary use of this code. Use a list comprehension instead of a generator comprehension. Just replace the parentheses with brackets. It evaluates to a list if you don't know.



                  array = [1, 2, 2, 4, 5]
                  f = [x for x in array if array.count(x) == 2]
                  array = [5, 6, 1, 2, 9]

                  print(f)
                  #[2, 2]


                  You are getting this response because of the nature of a generator. You're calling the generator when it't contents will evaluate to






                  share|improve this answer


























                  • Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.

                    – Suraj Kothari
                    8 hours ago











                  • With your change, list(f) becomes redundant.

                    – Mark Ransom
                    8 hours ago











                  • Lol @Mark Ransom, copy paste got me, I edited.

                    – Jaba
                    8 hours ago













                  • @SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!

                    – Jaba
                    8 hours ago
















                  2














                  You are not using a generator correctly if this is the primary use of this code. Use a list comprehension instead of a generator comprehension. Just replace the parentheses with brackets. It evaluates to a list if you don't know.



                  array = [1, 2, 2, 4, 5]
                  f = [x for x in array if array.count(x) == 2]
                  array = [5, 6, 1, 2, 9]

                  print(f)
                  #[2, 2]


                  You are getting this response because of the nature of a generator. You're calling the generator when it't contents will evaluate to






                  share|improve this answer


























                  • Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.

                    – Suraj Kothari
                    8 hours ago











                  • With your change, list(f) becomes redundant.

                    – Mark Ransom
                    8 hours ago











                  • Lol @Mark Ransom, copy paste got me, I edited.

                    – Jaba
                    8 hours ago













                  • @SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!

                    – Jaba
                    8 hours ago














                  2












                  2








                  2







                  You are not using a generator correctly if this is the primary use of this code. Use a list comprehension instead of a generator comprehension. Just replace the parentheses with brackets. It evaluates to a list if you don't know.



                  array = [1, 2, 2, 4, 5]
                  f = [x for x in array if array.count(x) == 2]
                  array = [5, 6, 1, 2, 9]

                  print(f)
                  #[2, 2]


                  You are getting this response because of the nature of a generator. You're calling the generator when it't contents will evaluate to






                  share|improve this answer















                  You are not using a generator correctly if this is the primary use of this code. Use a list comprehension instead of a generator comprehension. Just replace the parentheses with brackets. It evaluates to a list if you don't know.



                  array = [1, 2, 2, 4, 5]
                  f = [x for x in array if array.count(x) == 2]
                  array = [5, 6, 1, 2, 9]

                  print(f)
                  #[2, 2]


                  You are getting this response because of the nature of a generator. You're calling the generator when it't contents will evaluate to







                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited 8 hours ago

























                  answered 8 hours ago









                  JabaJaba

                  6,959175394




                  6,959175394













                  • Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.

                    – Suraj Kothari
                    8 hours ago











                  • With your change, list(f) becomes redundant.

                    – Mark Ransom
                    8 hours ago











                  • Lol @Mark Ransom, copy paste got me, I edited.

                    – Jaba
                    8 hours ago













                  • @SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!

                    – Jaba
                    8 hours ago



















                  • Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.

                    – Suraj Kothari
                    8 hours ago











                  • With your change, list(f) becomes redundant.

                    – Mark Ransom
                    8 hours ago











                  • Lol @Mark Ransom, copy paste got me, I edited.

                    – Jaba
                    8 hours ago













                  • @SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!

                    – Jaba
                    8 hours ago

















                  Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.

                  – Suraj Kothari
                  8 hours ago





                  Thank you. I seem to have used the wrong brackets. But in general using a generator comprehension seems odd.

                  – Suraj Kothari
                  8 hours ago













                  With your change, list(f) becomes redundant.

                  – Mark Ransom
                  8 hours ago





                  With your change, list(f) becomes redundant.

                  – Mark Ransom
                  8 hours ago













                  Lol @Mark Ransom, copy paste got me, I edited.

                  – Jaba
                  8 hours ago







                  Lol @Mark Ransom, copy paste got me, I edited.

                  – Jaba
                  8 hours ago















                  @SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!

                  – Jaba
                  8 hours ago





                  @SurajKothari It is not odd, it's a great tool! It just takes some time to wrap the ole brain around. Do some research you'll find that generators are amazing!

                  – Jaba
                  8 hours ago











                  0














                  Generator evaluation is "lazy" -- it doesn't get executed until you actualize it with a proper reference. With your line:



                  Look again at your output with the type of f: that object is a generator, not a sequence. It's waiting to be used, an iterator of sorts.



                  Your generator isn't evaluated until you start requiring values from it. At that point, it uses the available values at that point, not the point at which it was defined.





                  Code to "make it work"



                  That depends on what you mean by "make it work". If you want f to be a filtered list, then use a list, not a generator:



                  f = [x for x in array if array.count(x) == 2] # Filters original





                  share|improve this answer


























                  • I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.

                    – Suraj Kothari
                    8 hours ago
















                  0














                  Generator evaluation is "lazy" -- it doesn't get executed until you actualize it with a proper reference. With your line:



                  Look again at your output with the type of f: that object is a generator, not a sequence. It's waiting to be used, an iterator of sorts.



                  Your generator isn't evaluated until you start requiring values from it. At that point, it uses the available values at that point, not the point at which it was defined.





                  Code to "make it work"



                  That depends on what you mean by "make it work". If you want f to be a filtered list, then use a list, not a generator:



                  f = [x for x in array if array.count(x) == 2] # Filters original





                  share|improve this answer


























                  • I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.

                    – Suraj Kothari
                    8 hours ago














                  0












                  0








                  0







                  Generator evaluation is "lazy" -- it doesn't get executed until you actualize it with a proper reference. With your line:



                  Look again at your output with the type of f: that object is a generator, not a sequence. It's waiting to be used, an iterator of sorts.



                  Your generator isn't evaluated until you start requiring values from it. At that point, it uses the available values at that point, not the point at which it was defined.





                  Code to "make it work"



                  That depends on what you mean by "make it work". If you want f to be a filtered list, then use a list, not a generator:



                  f = [x for x in array if array.count(x) == 2] # Filters original





                  share|improve this answer















                  Generator evaluation is "lazy" -- it doesn't get executed until you actualize it with a proper reference. With your line:



                  Look again at your output with the type of f: that object is a generator, not a sequence. It's waiting to be used, an iterator of sorts.



                  Your generator isn't evaluated until you start requiring values from it. At that point, it uses the available values at that point, not the point at which it was defined.





                  Code to "make it work"



                  That depends on what you mean by "make it work". If you want f to be a filtered list, then use a list, not a generator:



                  f = [x for x in array if array.count(x) == 2] # Filters original






                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited 7 hours ago

























                  answered 8 hours ago









                  PrunePrune

                  43.2k143456




                  43.2k143456













                  • I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.

                    – Suraj Kothari
                    8 hours ago



















                  • I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.

                    – Suraj Kothari
                    8 hours ago

















                  I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.

                  – Suraj Kothari
                  8 hours ago





                  I somewhat understand. Could you show some code to make it work, because I need to re-assign the same list again in the main code.

                  – Suraj Kothari
                  8 hours ago











                  0














                  Generators are lazy and your newly defined array is used when you exhaust your generator after redefining. Therefore, the output is correct. A quick fix is to use a list comprehension by replacing parentheses () by brackets .



                  Moving on to how better to write your logic, counting a value in a loop has quadratic complexity. For an algorithm that works in linear time, you can use collections.Counter to count values, and keep a copy of your original list:



                  from collections import Counter

                  array = [1, 2, 2, 4, 5] # original array
                  counts = Counter(array) # count each value in array
                  old_array = array.copy() # make copy
                  array = [5, 6, 1, 2, 9] # updates array

                  # order relevant
                  res = [x for x in old_array if counts[x] >= 2]
                  print(res)
                  # [2, 2]

                  # order irrelevant
                  from itertools import chain
                  res = list(chain.from_iterable([x]*count for x, count in counts.items() if count >= 2))
                  print(res)
                  # [2, 2]


                  Notice the second version doesn't even require old_array and is useful if there is no need to maintain ordering of values in your original array.






                  share|improve this answer




























                    0














                    Generators are lazy and your newly defined array is used when you exhaust your generator after redefining. Therefore, the output is correct. A quick fix is to use a list comprehension by replacing parentheses () by brackets .



                    Moving on to how better to write your logic, counting a value in a loop has quadratic complexity. For an algorithm that works in linear time, you can use collections.Counter to count values, and keep a copy of your original list:



                    from collections import Counter

                    array = [1, 2, 2, 4, 5] # original array
                    counts = Counter(array) # count each value in array
                    old_array = array.copy() # make copy
                    array = [5, 6, 1, 2, 9] # updates array

                    # order relevant
                    res = [x for x in old_array if counts[x] >= 2]
                    print(res)
                    # [2, 2]

                    # order irrelevant
                    from itertools import chain
                    res = list(chain.from_iterable([x]*count for x, count in counts.items() if count >= 2))
                    print(res)
                    # [2, 2]


                    Notice the second version doesn't even require old_array and is useful if there is no need to maintain ordering of values in your original array.






                    share|improve this answer


























                      0












                      0








                      0







                      Generators are lazy and your newly defined array is used when you exhaust your generator after redefining. Therefore, the output is correct. A quick fix is to use a list comprehension by replacing parentheses () by brackets .



                      Moving on to how better to write your logic, counting a value in a loop has quadratic complexity. For an algorithm that works in linear time, you can use collections.Counter to count values, and keep a copy of your original list:



                      from collections import Counter

                      array = [1, 2, 2, 4, 5] # original array
                      counts = Counter(array) # count each value in array
                      old_array = array.copy() # make copy
                      array = [5, 6, 1, 2, 9] # updates array

                      # order relevant
                      res = [x for x in old_array if counts[x] >= 2]
                      print(res)
                      # [2, 2]

                      # order irrelevant
                      from itertools import chain
                      res = list(chain.from_iterable([x]*count for x, count in counts.items() if count >= 2))
                      print(res)
                      # [2, 2]


                      Notice the second version doesn't even require old_array and is useful if there is no need to maintain ordering of values in your original array.






                      share|improve this answer













                      Generators are lazy and your newly defined array is used when you exhaust your generator after redefining. Therefore, the output is correct. A quick fix is to use a list comprehension by replacing parentheses () by brackets .



                      Moving on to how better to write your logic, counting a value in a loop has quadratic complexity. For an algorithm that works in linear time, you can use collections.Counter to count values, and keep a copy of your original list:



                      from collections import Counter

                      array = [1, 2, 2, 4, 5] # original array
                      counts = Counter(array) # count each value in array
                      old_array = array.copy() # make copy
                      array = [5, 6, 1, 2, 9] # updates array

                      # order relevant
                      res = [x for x in old_array if counts[x] >= 2]
                      print(res)
                      # [2, 2]

                      # order irrelevant
                      from itertools import chain
                      res = list(chain.from_iterable([x]*count for x, count in counts.items() if count >= 2))
                      print(res)
                      # [2, 2]


                      Notice the second version doesn't even require old_array and is useful if there is no need to maintain ordering of values in your original array.







                      share|improve this answer












                      share|improve this answer



                      share|improve this answer










                      answered 5 hours ago









                      jppjpp

                      95.7k2157109




                      95.7k2157109























                          0














                          The root cause of the problem is that generators are lazy; variables are evaluated each time:



                          >>> l = [1, 2, 2, 4, 5, 5, 5]
                          >>> filtered = (x for x in l if l.count(x) == 2)
                          >>> l = [1, 2, 4, 4, 5, 6, 6]
                          >>> list(filtered)
                          [4]


                          It iterates over the original list and evaluates the condition with the current list. In this case, 4 appeared twice in the new list, causing it to appear in the result. It only appears once in the result because it only appeared once in the original list. The 6s appear twice in the new list, but never appear in the old list and are hence never shown.



                          Full function introspection for the curious (the line with the comment is the important line):



                          >>> l = [1, 2, 2, 4, 5]
                          >>> filtered = (x for x in l if l.count(x) == 2)
                          >>> l = [1, 2, 4, 4, 5, 6, 6]
                          >>> list(filtered)
                          [4]
                          >>> def f(original, new, count):
                          current = original
                          filtered = (x for x in current if current.count(x) == count)
                          current = new
                          return list(filtered)

                          >>> from dis import dis
                          >>> dis(f)
                          2 0 LOAD_FAST 0 (original)
                          3 STORE_DEREF 1 (current)

                          3 6 LOAD_CLOSURE 0 (count)
                          9 LOAD_CLOSURE 1 (current)
                          12 BUILD_TUPLE 2
                          15 LOAD_CONST 1 (<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>)
                          18 LOAD_CONST 2 ('f.<locals>.<genexpr>')
                          21 MAKE_CLOSURE 0
                          24 LOAD_DEREF 1 (current)
                          27 GET_ITER
                          28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                          31 STORE_FAST 3 (filtered)

                          4 34 LOAD_FAST 1 (new)
                          37 STORE_DEREF 1 (current)

                          5 40 LOAD_GLOBAL 0 (list)
                          43 LOAD_FAST 3 (filtered)
                          46 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                          49 RETURN_VALUE
                          >>> f.__code__.co_varnames
                          ('original', 'new', 'count', 'filtered')
                          >>> f.__code__.co_cellvars
                          ('count', 'current')
                          >>> f.__code__.co_consts
                          (None, <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>, 'f.<locals>.<genexpr>')
                          >>> f.__code__.co_consts[1]
                          <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>
                          >>> dis(f.__code__.co_consts[1])
                          3 0 LOAD_FAST 0 (.0)
                          >> 3 FOR_ITER 32 (to 38)
                          6 STORE_FAST 1 (x)
                          9 LOAD_DEREF 1 (current) # This loads the current list every time, as opposed to loading a constant.
                          12 LOAD_ATTR 0 (count)
                          15 LOAD_FAST 1 (x)
                          18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                          21 LOAD_DEREF 0 (count)
                          24 COMPARE_OP 2 (==)
                          27 POP_JUMP_IF_FALSE 3
                          30 LOAD_FAST 1 (x)
                          33 YIELD_VALUE
                          34 POP_TOP
                          35 JUMP_ABSOLUTE 3
                          >> 38 LOAD_CONST 0 (None)
                          41 RETURN_VALUE
                          >>> f.__code__.co_consts[1].co_consts
                          (None,)


                          To reiterate: The list to be iterated is only loaded once. Any closures in the condition or expression, however, are loaded from the enclosing scope each iteration. They are not stored in a constant.



                          The best solution for your problem would be to create a new variable referencing the original list and use that in your generator expression,.






                          share|improve this answer






























                            0














                            The root cause of the problem is that generators are lazy; variables are evaluated each time:



                            >>> l = [1, 2, 2, 4, 5, 5, 5]
                            >>> filtered = (x for x in l if l.count(x) == 2)
                            >>> l = [1, 2, 4, 4, 5, 6, 6]
                            >>> list(filtered)
                            [4]


                            It iterates over the original list and evaluates the condition with the current list. In this case, 4 appeared twice in the new list, causing it to appear in the result. It only appears once in the result because it only appeared once in the original list. The 6s appear twice in the new list, but never appear in the old list and are hence never shown.



                            Full function introspection for the curious (the line with the comment is the important line):



                            >>> l = [1, 2, 2, 4, 5]
                            >>> filtered = (x for x in l if l.count(x) == 2)
                            >>> l = [1, 2, 4, 4, 5, 6, 6]
                            >>> list(filtered)
                            [4]
                            >>> def f(original, new, count):
                            current = original
                            filtered = (x for x in current if current.count(x) == count)
                            current = new
                            return list(filtered)

                            >>> from dis import dis
                            >>> dis(f)
                            2 0 LOAD_FAST 0 (original)
                            3 STORE_DEREF 1 (current)

                            3 6 LOAD_CLOSURE 0 (count)
                            9 LOAD_CLOSURE 1 (current)
                            12 BUILD_TUPLE 2
                            15 LOAD_CONST 1 (<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>)
                            18 LOAD_CONST 2 ('f.<locals>.<genexpr>')
                            21 MAKE_CLOSURE 0
                            24 LOAD_DEREF 1 (current)
                            27 GET_ITER
                            28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                            31 STORE_FAST 3 (filtered)

                            4 34 LOAD_FAST 1 (new)
                            37 STORE_DEREF 1 (current)

                            5 40 LOAD_GLOBAL 0 (list)
                            43 LOAD_FAST 3 (filtered)
                            46 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                            49 RETURN_VALUE
                            >>> f.__code__.co_varnames
                            ('original', 'new', 'count', 'filtered')
                            >>> f.__code__.co_cellvars
                            ('count', 'current')
                            >>> f.__code__.co_consts
                            (None, <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>, 'f.<locals>.<genexpr>')
                            >>> f.__code__.co_consts[1]
                            <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>
                            >>> dis(f.__code__.co_consts[1])
                            3 0 LOAD_FAST 0 (.0)
                            >> 3 FOR_ITER 32 (to 38)
                            6 STORE_FAST 1 (x)
                            9 LOAD_DEREF 1 (current) # This loads the current list every time, as opposed to loading a constant.
                            12 LOAD_ATTR 0 (count)
                            15 LOAD_FAST 1 (x)
                            18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                            21 LOAD_DEREF 0 (count)
                            24 COMPARE_OP 2 (==)
                            27 POP_JUMP_IF_FALSE 3
                            30 LOAD_FAST 1 (x)
                            33 YIELD_VALUE
                            34 POP_TOP
                            35 JUMP_ABSOLUTE 3
                            >> 38 LOAD_CONST 0 (None)
                            41 RETURN_VALUE
                            >>> f.__code__.co_consts[1].co_consts
                            (None,)


                            To reiterate: The list to be iterated is only loaded once. Any closures in the condition or expression, however, are loaded from the enclosing scope each iteration. They are not stored in a constant.



                            The best solution for your problem would be to create a new variable referencing the original list and use that in your generator expression,.






                            share|improve this answer




























                              0












                              0








                              0







                              The root cause of the problem is that generators are lazy; variables are evaluated each time:



                              >>> l = [1, 2, 2, 4, 5, 5, 5]
                              >>> filtered = (x for x in l if l.count(x) == 2)
                              >>> l = [1, 2, 4, 4, 5, 6, 6]
                              >>> list(filtered)
                              [4]


                              It iterates over the original list and evaluates the condition with the current list. In this case, 4 appeared twice in the new list, causing it to appear in the result. It only appears once in the result because it only appeared once in the original list. The 6s appear twice in the new list, but never appear in the old list and are hence never shown.



                              Full function introspection for the curious (the line with the comment is the important line):



                              >>> l = [1, 2, 2, 4, 5]
                              >>> filtered = (x for x in l if l.count(x) == 2)
                              >>> l = [1, 2, 4, 4, 5, 6, 6]
                              >>> list(filtered)
                              [4]
                              >>> def f(original, new, count):
                              current = original
                              filtered = (x for x in current if current.count(x) == count)
                              current = new
                              return list(filtered)

                              >>> from dis import dis
                              >>> dis(f)
                              2 0 LOAD_FAST 0 (original)
                              3 STORE_DEREF 1 (current)

                              3 6 LOAD_CLOSURE 0 (count)
                              9 LOAD_CLOSURE 1 (current)
                              12 BUILD_TUPLE 2
                              15 LOAD_CONST 1 (<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>)
                              18 LOAD_CONST 2 ('f.<locals>.<genexpr>')
                              21 MAKE_CLOSURE 0
                              24 LOAD_DEREF 1 (current)
                              27 GET_ITER
                              28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                              31 STORE_FAST 3 (filtered)

                              4 34 LOAD_FAST 1 (new)
                              37 STORE_DEREF 1 (current)

                              5 40 LOAD_GLOBAL 0 (list)
                              43 LOAD_FAST 3 (filtered)
                              46 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                              49 RETURN_VALUE
                              >>> f.__code__.co_varnames
                              ('original', 'new', 'count', 'filtered')
                              >>> f.__code__.co_cellvars
                              ('count', 'current')
                              >>> f.__code__.co_consts
                              (None, <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>, 'f.<locals>.<genexpr>')
                              >>> f.__code__.co_consts[1]
                              <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>
                              >>> dis(f.__code__.co_consts[1])
                              3 0 LOAD_FAST 0 (.0)
                              >> 3 FOR_ITER 32 (to 38)
                              6 STORE_FAST 1 (x)
                              9 LOAD_DEREF 1 (current) # This loads the current list every time, as opposed to loading a constant.
                              12 LOAD_ATTR 0 (count)
                              15 LOAD_FAST 1 (x)
                              18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                              21 LOAD_DEREF 0 (count)
                              24 COMPARE_OP 2 (==)
                              27 POP_JUMP_IF_FALSE 3
                              30 LOAD_FAST 1 (x)
                              33 YIELD_VALUE
                              34 POP_TOP
                              35 JUMP_ABSOLUTE 3
                              >> 38 LOAD_CONST 0 (None)
                              41 RETURN_VALUE
                              >>> f.__code__.co_consts[1].co_consts
                              (None,)


                              To reiterate: The list to be iterated is only loaded once. Any closures in the condition or expression, however, are loaded from the enclosing scope each iteration. They are not stored in a constant.



                              The best solution for your problem would be to create a new variable referencing the original list and use that in your generator expression,.






                              share|improve this answer















                              The root cause of the problem is that generators are lazy; variables are evaluated each time:



                              >>> l = [1, 2, 2, 4, 5, 5, 5]
                              >>> filtered = (x for x in l if l.count(x) == 2)
                              >>> l = [1, 2, 4, 4, 5, 6, 6]
                              >>> list(filtered)
                              [4]


                              It iterates over the original list and evaluates the condition with the current list. In this case, 4 appeared twice in the new list, causing it to appear in the result. It only appears once in the result because it only appeared once in the original list. The 6s appear twice in the new list, but never appear in the old list and are hence never shown.



                              Full function introspection for the curious (the line with the comment is the important line):



                              >>> l = [1, 2, 2, 4, 5]
                              >>> filtered = (x for x in l if l.count(x) == 2)
                              >>> l = [1, 2, 4, 4, 5, 6, 6]
                              >>> list(filtered)
                              [4]
                              >>> def f(original, new, count):
                              current = original
                              filtered = (x for x in current if current.count(x) == count)
                              current = new
                              return list(filtered)

                              >>> from dis import dis
                              >>> dis(f)
                              2 0 LOAD_FAST 0 (original)
                              3 STORE_DEREF 1 (current)

                              3 6 LOAD_CLOSURE 0 (count)
                              9 LOAD_CLOSURE 1 (current)
                              12 BUILD_TUPLE 2
                              15 LOAD_CONST 1 (<code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>)
                              18 LOAD_CONST 2 ('f.<locals>.<genexpr>')
                              21 MAKE_CLOSURE 0
                              24 LOAD_DEREF 1 (current)
                              27 GET_ITER
                              28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                              31 STORE_FAST 3 (filtered)

                              4 34 LOAD_FAST 1 (new)
                              37 STORE_DEREF 1 (current)

                              5 40 LOAD_GLOBAL 0 (list)
                              43 LOAD_FAST 3 (filtered)
                              46 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                              49 RETURN_VALUE
                              >>> f.__code__.co_varnames
                              ('original', 'new', 'count', 'filtered')
                              >>> f.__code__.co_cellvars
                              ('count', 'current')
                              >>> f.__code__.co_consts
                              (None, <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>, 'f.<locals>.<genexpr>')
                              >>> f.__code__.co_consts[1]
                              <code object <genexpr> at 0x02DD36B0, file "<pyshell#17>", line 3>
                              >>> dis(f.__code__.co_consts[1])
                              3 0 LOAD_FAST 0 (.0)
                              >> 3 FOR_ITER 32 (to 38)
                              6 STORE_FAST 1 (x)
                              9 LOAD_DEREF 1 (current) # This loads the current list every time, as opposed to loading a constant.
                              12 LOAD_ATTR 0 (count)
                              15 LOAD_FAST 1 (x)
                              18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
                              21 LOAD_DEREF 0 (count)
                              24 COMPARE_OP 2 (==)
                              27 POP_JUMP_IF_FALSE 3
                              30 LOAD_FAST 1 (x)
                              33 YIELD_VALUE
                              34 POP_TOP
                              35 JUMP_ABSOLUTE 3
                              >> 38 LOAD_CONST 0 (None)
                              41 RETURN_VALUE
                              >>> f.__code__.co_consts[1].co_consts
                              (None,)


                              To reiterate: The list to be iterated is only loaded once. Any closures in the condition or expression, however, are loaded from the enclosing scope each iteration. They are not stored in a constant.



                              The best solution for your problem would be to create a new variable referencing the original list and use that in your generator expression,.







                              share|improve this answer














                              share|improve this answer



                              share|improve this answer








                              edited 3 hours ago

























                              answered 4 hours ago









                              Solomon UckoSolomon Ucko

                              6271719




                              6271719























                                  0














                                  Others have already explained the root cause of the issue - the generator is binding to the name of the array local variable, rather than its value.



                                  The most pythonic solution is definitely the list comprehension:



                                  f = [x for x in array if array.count(x) == 2]




                                  However, if there is some reason that you don't want to create a list, you can also force a scope close over array:



                                  f = (lambda array=array: (x for x in array if array.count(x) == 2))()


                                  What's happening here is that the lambda captures the reference to array at the time the line is run, ensuring that the generator sees the variable you expect, even if the variable is later redefined.



                                  Note that this still binds to the variable (reference), not the value, so, for example, the following will print [2, 2, 4, 4]:



                                  array = [1, 2, 2, 4, 5] # Original array

                                  f = (lambda array=array: (x for x in array if array.count(x) == 2))() # Close over array
                                  array.append(4) # This *will* be captured

                                  array = [5, 6, 1, 2, 9] # Updates original to something else

                                  print(list(f)) # Outputs [2, 2, 4, 4]




                                  This is a common pattern in some languages, but it's not very pythonic, so only really makes sense if there's a very good reason for not using the list comprehension (e.g., if array is very long, or is being used in a nested generator comprehension, and you're concerned about memory).






                                  share|improve this answer




























                                    0














                                    Others have already explained the root cause of the issue - the generator is binding to the name of the array local variable, rather than its value.



                                    The most pythonic solution is definitely the list comprehension:



                                    f = [x for x in array if array.count(x) == 2]




                                    However, if there is some reason that you don't want to create a list, you can also force a scope close over array:



                                    f = (lambda array=array: (x for x in array if array.count(x) == 2))()


                                    What's happening here is that the lambda captures the reference to array at the time the line is run, ensuring that the generator sees the variable you expect, even if the variable is later redefined.



                                    Note that this still binds to the variable (reference), not the value, so, for example, the following will print [2, 2, 4, 4]:



                                    array = [1, 2, 2, 4, 5] # Original array

                                    f = (lambda array=array: (x for x in array if array.count(x) == 2))() # Close over array
                                    array.append(4) # This *will* be captured

                                    array = [5, 6, 1, 2, 9] # Updates original to something else

                                    print(list(f)) # Outputs [2, 2, 4, 4]




                                    This is a common pattern in some languages, but it's not very pythonic, so only really makes sense if there's a very good reason for not using the list comprehension (e.g., if array is very long, or is being used in a nested generator comprehension, and you're concerned about memory).






                                    share|improve this answer


























                                      0












                                      0








                                      0







                                      Others have already explained the root cause of the issue - the generator is binding to the name of the array local variable, rather than its value.



                                      The most pythonic solution is definitely the list comprehension:



                                      f = [x for x in array if array.count(x) == 2]




                                      However, if there is some reason that you don't want to create a list, you can also force a scope close over array:



                                      f = (lambda array=array: (x for x in array if array.count(x) == 2))()


                                      What's happening here is that the lambda captures the reference to array at the time the line is run, ensuring that the generator sees the variable you expect, even if the variable is later redefined.



                                      Note that this still binds to the variable (reference), not the value, so, for example, the following will print [2, 2, 4, 4]:



                                      array = [1, 2, 2, 4, 5] # Original array

                                      f = (lambda array=array: (x for x in array if array.count(x) == 2))() # Close over array
                                      array.append(4) # This *will* be captured

                                      array = [5, 6, 1, 2, 9] # Updates original to something else

                                      print(list(f)) # Outputs [2, 2, 4, 4]




                                      This is a common pattern in some languages, but it's not very pythonic, so only really makes sense if there's a very good reason for not using the list comprehension (e.g., if array is very long, or is being used in a nested generator comprehension, and you're concerned about memory).






                                      share|improve this answer













                                      Others have already explained the root cause of the issue - the generator is binding to the name of the array local variable, rather than its value.



                                      The most pythonic solution is definitely the list comprehension:



                                      f = [x for x in array if array.count(x) == 2]




                                      However, if there is some reason that you don't want to create a list, you can also force a scope close over array:



                                      f = (lambda array=array: (x for x in array if array.count(x) == 2))()


                                      What's happening here is that the lambda captures the reference to array at the time the line is run, ensuring that the generator sees the variable you expect, even if the variable is later redefined.



                                      Note that this still binds to the variable (reference), not the value, so, for example, the following will print [2, 2, 4, 4]:



                                      array = [1, 2, 2, 4, 5] # Original array

                                      f = (lambda array=array: (x for x in array if array.count(x) == 2))() # Close over array
                                      array.append(4) # This *will* be captured

                                      array = [5, 6, 1, 2, 9] # Updates original to something else

                                      print(list(f)) # Outputs [2, 2, 4, 4]




                                      This is a common pattern in some languages, but it's not very pythonic, so only really makes sense if there's a very good reason for not using the list comprehension (e.g., if array is very long, or is being used in a nested generator comprehension, and you're concerned about memory).







                                      share|improve this answer












                                      share|improve this answer



                                      share|improve this answer










                                      answered 17 mins ago









                                      sapisapi

                                      6,57963161




                                      6,57963161






























                                          draft saved

                                          draft discarded




















































                                          Thanks for contributing an answer to Stack Overflow!


                                          • Please be sure to answer the question. Provide details and share your research!

                                          But avoid



                                          • Asking for help, clarification, or responding to other answers.

                                          • Making statements based on opinion; back them up with references or personal experience.


                                          To learn more, see our tips on writing great answers.




                                          draft saved


                                          draft discarded














                                          StackExchange.ready(
                                          function () {
                                          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54245618%2funexpected-behaviour-with-python-generator%23new-answer', 'question_page');
                                          }
                                          );

                                          Post as a guest















                                          Required, but never shown





















































                                          Required, but never shown














                                          Required, but never shown












                                          Required, but never shown







                                          Required, but never shown

































                                          Required, but never shown














                                          Required, but never shown












                                          Required, but never shown







                                          Required, but never shown







                                          Popular posts from this blog

                                          Liste der Baudenkmale in Friedland (Mecklenburg)

                                          Single-Malt-Whisky

                                          Czorneboh