[ Modifying a shared dict object among processes doesn't work ]

I think i found a similar type of question here already , but did not find a good explanation which i could understand easily. I tried to read the python doc as well but didn't find a detailed explanation there as well.

Now here is the code :

from multiprocessing import Process, Manager

def f(d):
    d[1] = [[1]] 
    if 2 in d.keys():
       d[2].append([2])
    else:
       d[2] = [[2]]
    d[3] = [[3]]

if __name__ == '__main__':

    with Manager() as manager:
        d = manager.dict()

        p1 = Process(target=f, args=(d,))
        p2 = Process(target=f, args=(d,))
        p1.start()
        p2.start
        p1.join()
        p2.join

        print(d)

I have two process running and sharing a dict object from Manager process. I can add new keys-value pair to the dict object but cannot modify the values if the key already present.

Why is this so? What are the problems which can arise if we use such shared objects if want to modify?

Answer 1

First, you didn't start the second process.

p2.start   # <--- no `()`
p2.join    # <--- no `()`

Second, according to the multiprocessing.managers.SyncManager documentation NOTE:

NOTE: Modifications to mutable values or items in dict and list proxies will not be propagated through the manager, because the proxy has no way of knowing when its values or items are modified. To modify such an item, you can re-assign the modified object to the container proxy.

So, your program should looks like:

from multiprocessing import Process, Manager

def f(d):
    d[1] = [[1]] 
    if 2 in d:  # NOTE: You don't need to use `keys()`. `dict` support `in` operator.
       d[2] = d[2] + [[2]]
    else:
       d[2] = [[2]]
    d[3] = [[3]]

if __name__ == '__main__':

    with Manager() as manager:
        d = manager.dict()

        p1 = Process(target=f, args=(d,))
        p2 = Process(target=f, args=(d,))
        p1.start()
        p2.start()
        p1.join()
        p2.join()

        print(d)

But, there's another concern. The following statement involve multiple reads, write to the shared object; which is not atomic; So there's no guarantee that above program yield same result every time.

if 2 in d:
   d[2] = d[2] + [[2]]
else:
   d[2] = [[2]]

You need to use a lock to make sure the shared object is access by only one process.