Memory of copied PIL Images is not released #7935

Activity

1. radarhere

   added

   Memory

   on Apr 2, 2024
2. wiredfool commented on Apr 2, 2024

   

   wiredfool

   on Apr 2, 2024

   Last edited by wiredfool

   Member

   More actions

   Pillow's memory allocator doesn't necessarily release the memory in the pool back as soon as an image is destroyed, as it uses that memory pool for future allocations. See Storage.c (https://github.com/python-pillow/Pillow/blob/main/src/libImaging/Storage.c#L310) for the implementation.

   If you repeatedly open and close an image, you should not see the memory increase, but it won't necessarily drop between destruction and allocation again.

   (edit: related: #5401, #3610)
3. Yay295 commented on Apr 2, 2024

   

   Yay295

   on Apr 2, 2024

   Contributor

   More actions

   It looks like it caches 0 blocks by default though.

   Pillow/src/libImaging/Storage.c

   Lines 260 to 271 in aeeb596

   	struct ImagingMemoryArena ImagingDefaultArena = {
   	1, // alignment
   	16 * 1024 * 1024, // block_size
   	0, // blocks_max
   	0, // blocks_cached
   	NULL, // blocks_pool
   	0,
   	0,
   	0,
   	0,
   	0 // Stats
   	};

   And you can set the number of blocks to cache with the PILLOW_BLOCKS_MAX environment variable.

   Pillow/src/PIL/Image.py

   Lines 3624 to 3656 in aeeb596

   	def _apply_env_variables(env=None) -> None:
   	if env is None:
   	env = os.environ
   	for var_name, setter in [
   	("PILLOW_ALIGNMENT", core.set_alignment),
   	("PILLOW_BLOCK_SIZE", core.set_block_size),
   	("PILLOW_BLOCKS_MAX", core.set_blocks_max),
   	]:
   	if var_name not in env:
   	continue
   	var = env[var_name].lower()
   	units = 1
   	for postfix, mul in [("k", 1024), ("m", 1024 * 1024)]:
   	if var.endswith(postfix):
   	units = mul
   	var = var[: -len(postfix)]
   	try:
   	var = int(var) * units
   	except ValueError:
   	warnings.warn(f"{var_name} is not int")
   	continue
   	try:
   	setter(var)
   	except ValueError as e:
   	warnings.warn(f"{var_name}: {e}")
   	_apply_env_variables()

   There's a docs page for this actually: https://pillow.readthedocs.io/en/stable/reference/block_allocator.html
4. TTMaDe commented on Apr 3, 2024

   

   TTMaDe

   on Apr 3, 2024

   Author

   More actions

   Thanks for the quick answers!

   Indeed, when I set the PILLOW_BLOCKS_MAX=5 environment variable the used memory decreases when releasing/closing the images. But after reading the linked docs page, I would expect that if I manually set the environment variable to 0 (or just leave it unset), the memory pools will be disabled, no block caching will occur and the memory of closed images will be freed immediately.
   But with this default settings our application ran out of memory on a 16 GB Linux system after reading, modifying and closing images in a loop.

   👍React with 👍1Reacted by Alberto Agudo
5. radarhere commented on Apr 3, 2024

   

   radarhere

   on Apr 3, 2024

   Last edited by radarhere

   Member

   More actions

   This may or may not help - in your original code you open an image and don't close it. It is recommended instead that you either call img.close() when you are done or use a context manager for the image. See https://pillow.readthedocs.io/en/stable/deprecations.html#image-del and

   Pillow/src/PIL/Image.py

   Lines 560 to 565 in e8ab564

   	def close(self) -> None:
   	"""
   	Closes the file pointer, if possible.
   	This operation will destroy the image core and release its memory.
   	The image data will be unusable afterward.

   Edit: I see you've mentioned 'closing images' in your comments, so this remark can just be for reference to others.
6. wiredfool commented on Apr 3, 2024

   

   wiredfool

   on Apr 3, 2024

   Member

   More actions

   I didn't catch this before but what you're doing is basically opening 50 copies of an image and keeping them all.

   Can you show us a flow where you expect constant memory usage?
7. TTMaDe commented on Apr 3, 2024

   

   TTMaDe

   on Apr 3, 2024

   Author

   More actions

   Yes, in the first loop I open the image 50 times and hold 50 copies so the memory usage increases which is ok.
   In the second loop, I delete a container containing an image copy in each iteration, so I would expect memory usage to decrease after each iteration. But the used memory only decreased if I manually set PILLOW_BLOCKS_MAX to a value > 0.

   Manually closing the image copy via self.value.close() in the destructor of the Container class doesn't make a difference so I removed it:

   import gc
   import os
   import sys
   import time
   
   import PIL.Image
   import psutil
   
   FILE = './test_image.png'
   
   
   def LogMemory():
       pid = os.getpid()
       rss = 0
       for mmap in psutil.Process(pid).memory_maps():
           # All memory that this process holds in RAM. RSS = USS + Shared.
           rss += mmap.rss
       return rss
   
   
   class Container:
   
       def __init__(self):
           self.value = None
   
       def SetValue(self, value):
           self.value = self._copyValue(value)
   
       def GetValue(self):
           return self._copyValue(self.value)
   
       def _copyValue(self, value):
           return value.copy()
   
   
   if __name__ == '__main__':
   
       print(f'Using Python {sys.version}, PIL {PIL.__version__}')
       print(f'PILLOW_ALIGNMENT: {PIL.Image.core.get_alignment()}')
       print(f'PILLOW_BLOCK_SIZE: {PIL.Image.core.get_block_size()}')
       print(f'PILLOW_BLOCKS_MAX: {PIL.Image.core.get_blocks_max()}')
   
       containers = []
   
       for i in range(50):
           before = LogMemory()
           img = PIL.Image.open(FILE)
           # img = [1] * (1920*1080*3)  # this works!
           container = Container()
           container.SetValue(img)
           containers.append(container)
           after = LogMemory()
           print(f'Loaded image {i} took {after-before} bytes')
   
       for i in range(len(containers)):
           before = LogMemory()
           containers.pop()
           time.sleep(0.1)
           after = LogMemory()
           print(f'popped container {i} released {before-after} bytes')
   
       print('Delete list')
       before = LogMemory()
       containers = None
       gc.collect()
       after = LogMemory()
       print(f'Finally released {before-after} bytes')

   Running the code with PILLOW_BLOCKS_MAX=1 prints:

   Using Python 3.11.8 (main, Feb 25 2024, 16:41:26) [GCC 9.4.0], PIL 10.2.0
   PILLOW_ALIGNMENT: 1
   PILLOW_BLOCK_SIZE: 16777216
   PILLOW_BLOCKS_MAX: 1
   Loaded image 0 took 17731584 bytes
   Loaded image 1 took 8339456 bytes
   Loaded image 2 took 8298496 bytes
   Loaded image 3 took 8298496 bytes
   ...
   Loaded image 49 took 8298496 bytes
   popped container 0 released -12288 bytes
   popped container 1 released 0 bytes
   popped container 2 released 8298496 bytes
   popped container 3 released 8298496 bytes
   popped container 4 released 8298496 bytes
   popped container 5 released 8298496 bytes
   popped container 6 released 8298496 bytes
   popped container 7 released 8298496 bytes
   ...
   popped container 48 released 8298496 bytes
   popped container 49 released 8298496 bytes
   Delete list
   Finally released 0 bytes
   

   The memory usage decreases with every containers.pop()

   But when I run with PILLOW_BLOCKS_MAX=0 or just leave the environment variable unset I get:

   Using Python 3.11.8 (main, Feb 25 2024, 16:41:26) [GCC 9.4.0], PIL 10.2.0
   PILLOW_ALIGNMENT: 1
   PILLOW_BLOCK_SIZE: 16777216
   PILLOW_BLOCKS_MAX: 0
   Loaded image 0 took 17735680 bytes
   Loaded image 1 took 8114176 bytes
   Loaded image 2 took 8069120 bytes
   Loaded image 3 took 8036352 bytes
   Loaded image 4 took 8044544 bytes
   Loaded image 5 took 8052736 bytes
   Loaded image 6 took 8052736 bytes
   ...
   Loaded image 48 took 8052736 bytes
   Loaded image 49 took 8065024 bytes
   popped container 0 released 0 bytes
   popped container 1 released 0 bytes
   popped container 2 released 0 bytes
   popped container 3 released 0 bytes
   popped container 4 released 0 bytes
   popped container 5 released 0 bytes
   popped container 6 released 0 bytes
   popped container 7 released 0 bytes
   ...
   popped container 48 released 0 bytes
   popped container 49 released 8298496 bytes
   Delete list
   Finally released 0 bytes
   

   and the used memory doesn't decrease while popping the containers from the list.

   So setting PILLOW_BLOCKS_MAX to a value > 0 fixes my problem because the memory is freed but after reading the linked doc I would expect setting PILLOW_BLOCKS_MAX to 0 disables caches and memory will also be freed on each iteration.

   ❤️React with ❤️1Reacted by Alberto Agudo

   👀React with 👀1Reacted by Rafael Weingartner-Ortner
8. RafaelWO commented on Nov 18, 2024

   

   RafaelWO

   on Nov 18, 2024

   Last edited by RafaelWO

   More actions

   I have a similar problem when loading and rotating around 2k images. My code is similar to the following:

   frames: list[PIL.Image.Image] = load_images(...)
   # After loading the frames, my memory usage is under 500MB
   
   for idx, frame in enumerate(frames):
       # This is were a lot of memory is allocated
       rotated_frame = frame.rotate(-90, expand=True)
       rotated_frame.save(work_dir / f"{idx:0>4d}.jpeg")
       # I expect the memory to be freed for ever loop iteration (per image)

   Below is a flamegraph of my program created with memray. As you can see, the memory is not freed and reaches over 6GB at the end.

   
9. wiredfool commented on Nov 18, 2024

   

   wiredfool

   on Nov 18, 2024

   Last edited by radarhere

   Member

   More actions

   @RafaelWO

   I think you've got something else going on, depending on what's in load_images.

   If you do this:

   frames: list[PIL.Image.Image] = load_images(...)
   
   for idx, frame in enumerate(frames):
       frame.load()

   Does the memory profile look the same?

   👀React with 👀1Reacted by Rafael Weingartner-Ortner
10. RafaelWO commented on Nov 18, 2024

    

    RafaelWO

    on Nov 18, 2024

    More actions

    @wiredfool

    I think you've got something else going on, depending on what's in load_images.

    Essentially, load_images(...) downloads data (read: bytes) HTTP GET and creates and image via PIL.Image.open(io.BytesIO(data)) for every image.

    If you do this (...) Does the memory profile look the same?

    Good point, it looks the same:

    

    So I guess this means that PIL.Image.open is lazy and the image is only loaded if you first "touch" it?

    ---

    Either way, it seems that I can only mitigate the high memory usage by not having all images in a list, right?
11. RafaelWO commented on Nov 18, 2024

    

    RafaelWO

    on Nov 18, 2024

    Last edited by RafaelWO

    More actions

    Either way, it seems that I can only mitigate the high memory usage by not having all images in a list, right?

    I was able to bring down the memory usage to ~500MB by calling .close() on the frames in the loop:

    frames: list[PIL.Image.Image] = load_images(...)
    
    for idx, frame in enumerate(frames):
        rotated_frame = frame.rotate(-90, expand=True)
        rotated_frame.save(work_dir / f"{idx:0>4d}.jpeg")
    
        # UPDATE: Free the memory
        frame.close()
        rotated_frame.close()

    Click for memory profile

    Old	New

    Thanks for the clarification and your help, @wiredfool ! 🙂
12. wiredfool commented on Nov 18, 2024

    

    wiredfool

    on Nov 18, 2024

    Member

    More actions

    The other thing you could to is make load images return a generator instead of a list, so that you're only actually opening one image at a time.

    👍React with 👍1Reacted by Rafael Weingartner-Ortner
13. TTMaDe commented on Nov 22, 2024

    

    TTMaDe

    on Nov 22, 2024

    Author

    More actions

    Sorry, I also have to dig out the topic again because we still have some problems on Linux.

    I created this simple script which loads an image (4000x2250 pixels) 100 times and saves a copy to a list. Afterwards the elements are popped from the list one after another. On Windows the used memory for the process is decreasing slowly while the image copies are deleted in the second loop (as intended). On Linux the memory is still in use and released when the Python process is finished:

    import gc
    import time
    
    import PIL
    from PIL import Image
    
    FILE = './test_image_medium.png'
    
    if __name__ == '__main__':
        print(f'Using Pillow version: {PIL.__version__} from {PIL.__file__}')
        buffer = []
    
        for i in range(100):
            with Image.open(FILE) as img:
                img.load()  # open is lazy operation
                buffer.append(img.copy())
                print(f'Loaded image {i}')
    
        for i in range(len(buffer)):
            item = buffer.pop()
            # item.close()
            print(f'Popped {i}')
            gc.collect()
            time.sleep(0.1)
    
        print('Delete buffer')
        del buffer
        time.sleep(1)
        gc.collect()
        time.sleep(5)
        print('Done')

    The really weird things is: When I delete the .venv/lib/python3.12/site-packages/PIL/__pycache__/ directory with all the *.pyc files in my Linux Python environment, the script will work as intended and the memory is freed in the second loop. When I run the same script again, the problem occurs again until I delete the pyc files.

    Could this be a problem of the CPython byte code compiler or does Pillow some special stuff when loaded from the *.pyc byte code?
    I tested with Python 3.11 and Python 3.12 on Ubuntu 20.04 (WSL) and Pillow 11.0.0 from pypi (same behavior when I compile the lib myself).

    👀React with 👀5Reacted by Rafael Weingartner-Ortner, Brandon Fan, Devin Dahoon Kim, Alberto Agudo and kocsszi
14. alberto-agudo commented on Jun 27, 2025

    

    alberto-agudo

    on Jun 27, 2025

    More actions

    So setting PILLOW_BLOCKS_MAX to a value > 0 fixes my problem because the memory is freed but after reading the linked doc I would expect setting PILLOW_BLOCKS_MAX to 0 disables caches and memory will also be freed on each iteration.

    Can confirm that I'm experiencing the same problem under the following system and library configurations.

    pillow==11.2.1
    psutil==7.0.0
    Python 3.12.11
    OS: Debian GNU/Linux 12 (bookworm)
    

    Setting PILLOW_BLOCKS_MAX to > 0 also allows memory to be freed after deleting/closing images, but that's not the behavior you'd expect based on the aforementioned documentation (=0 should be caching 0 blocks).
15. aclark4life

    added

    PNG

    on Oct 8, 2026