Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

can you explain how generators work with multiprocess (Thread based pool) ?

is ps internal variable unique for each Thread or same?

is it safe to execute your primes() from different threads?



> can you explain how generators work with multiprocess

The best way to think of a generator is as an object implementing the iteration protocol. They don't really interact with concurrency, as far as multiprocess is concerned, they're just regular objects. So the answer is that it depends on how you plan to share memory between the processes.

> is ps internal variable unique for each Thread or same?

ps is local to the generator instance.

  def f():
      x = 0
      while True:
          yield (x := x + 1)
 
  >>> f()
  <generator object f at 0x10412e500>
  >>> x = f()
  >>> y = f()
  >>> next(x)
  1
  >>> next(x)
  2
  >>> next(y)
  1
> is it safe to execute your primes() from different threads?

For this specific generator, you would run into the GIL. More generally, if you're talking about non CPU-bound operations, you need to synchronize the threads. It's worth looking into asyncio for those use cases.


A yield will simply return a generator object, which contains information about the next value to use, and how to continue the function execution. That's why you need to use functions that yield things inside loops or list(...).

If you run it from different threads I guess it will be the same as calling the function multiple times, it will return a new started-from-the-top generator.

    def sum():
        yield 1
        yield 2
    print(repr(sum()))
    print(next(sum()))
    print(next(sum()))
Prints

    <generator object sum at 0x7fc6f14823c0>
    1
    1


so Thread based based pool will have same instance of generator, while Process based pool with have unique instance of generator?


In this example, calling sum() creates a generator and returns it. Say g = sum(). If you share g between threads, they will all use the same generator object! If you call sum() separately per thread, they will be different generators.

If you try to send g to a different process, you will get an error, because it doesn't serialize.


I don't know if a generator can be shared across threads, but in that case ... I have no idea :/

You'll need to search, or try!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: