cmueller/ansible
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Stream image data in load_image() to avoid out-of-memory

Browse source 
Reading the entire tar file into memory can result in out-of-memory
conditions such as this traceback:

Traceback (most recent call last):
  File "/tmp/ansible_YELTSu/ansible_module_docker_image.py", line 486, in load_image
    self.client.load_image(image_data)
  File "/usr/local/lib/python2.7/dist-packages/docker/api/image.py", line 147, in load_image
    res = self._post(self._url("/images/load"), data=data)
  ...
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 848, in _send_output
    msg += message_body
MemoryError

Luckily docker-py's load_image(), which calls requests post(), accepts a
file-like object instead of a string.  Pass in the file object to avoid
reading the full file into memory.  This allows larger tar files to load
succesfully.

Signed-off-by: Stefan Hajnoczi <stefanha@gmail.com>
This commit is contained in:
Stefan Hajnoczi 2016-06-08 23:50:44 +01:00
parent a8072f9ef0
commit 47cc422b8a
1 changed files with 8 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
cloud/docker/docker_image.py
View file

@ -474,19 +474,22 @@ class ImageManager(DockerBaseClass):
:return: image dict
'''
try:
self.log("Reading image data from %s" % self.load_path)
self.log("Opening image %s" % self.load_path)
image_tar = open(self.load_path, 'r')
image_data = image_tar.read()
image_tar.close()
except Exception as exc:
self.fail("Error reading image data %s - %s" % (self.load_path, str(exc)))
self.fail("Error opening image %s - %s" % (self.load_path, str(exc)))
try:
self.log("Loading image from %s" % self.load_path)
self.client.load_image(image_data)
self.client.load_image(image_tar)
except Exception as exc:
self.fail("Error loading image %s - %s" % (self.name, str(exc)))
try:
image_tar.close()
except Exception as exc:
self.fail("Error closing image %s - %s" % (self.name, str(exc)))
return self.client.find_image(self.name, self.tag)