Talk Submission

If you are interested in attending this talk at PyCon JP 2016, please use the social media share buttons below. We will consider the popularity of the proposals when making our selection.

talk

Sleepy: How to suspend and resume your cpython process from inside(en)

Speakers

David Weil

Audience level:

Experienced

Category:

Cloud

Description

In this talk we'll propose and explain how it is possible suspend & resume a python process execution from within python using no external-help. Pure-python-ffi-free code. This kind of tool is useful with the proliferation of cloud serv since they offer restricted execution env in terms of time/mem and process relocation/prio is required. Besides we’ll get unexpected debugger improvements.

Objectives

-attendees will be introduced to the problem of suspending/resuming a python process and where it could be useful (cloud computing, process distribution, multi-platform code execution) -will learn why code state serialization isn't possible in a straight-forward way. Some C objects can't be serialized! -a method enabling suspend and resume python code is developed and the natural problem found will be overcome using python standard lib. (and a pure-python lib) tools in a hardcore approach: -we'll see why python's debugger is THE (almost-ready) tool for our job. -we'll present python's basic flow-control AST and introduce variations which will overcome debugger's limitations -we'll show how to use pycodegen to rewrite python's byte-code -present attendees a way to make python work as they would like it to

Abstract

Detailed Abstract ---------------------- People will be introduced to a simple case of what continuations are, i.e.: suspending a whole python process and resuming it from where it was saved. We'll present some scenarios where it would be useful: * restricted time execution environments * distributed computing farms * tasks would become dynamic, meaning they could be switched from one host to another at the user's will -run- time code problems evaluation by code re-execution, etc. A comparison to different alternatives will be shown: * other languages features * operating system provided features We'll present a method to persist execution state and then explain how to restore it. To do that, we use python's debugger in order to restore the state. Some problems arise in that process such as: * how to persist and restore objects created in C * how to restore process execution using debugger's functions To persist and restore objects created in C, I created python wrappers which would replicate C-created objects and expose their own methods and instance variables. To restore a process execution in a given point, we use the python debugger. With it, is possible to start the code in a programmatic way instrumenting to jump directly into the function call belonging to frames present in saved call-stack. The main problem here lies in the fact that the debugger is not powerful enough to jump from any function entrypoint into arbitrary points inside the funcion, i.e.: you can’t jump inside a block of code! For that, I present 3 different approaches to workaround that: 1. block-unrolling : ie: pasting and compiling block’s code in the same place it is located and also, after function end, in an unrolled way (check added block1 after return) : for something in some_iterator: block1 return block1 2. the alternative is to replace the expressions which control loops with assisted functions, like: for something in my_library_controlled_iterator(original_iterator): block1 return letting us control the debugger and execute the loop instructions (returning controlled value from the assisted function iterator), and then jump again to call the function for the next frame in the call-stack. 3. it is possible to rewrite python’s byte-code to have no block byte-codes. This is the method I chose for my implementation, and I will explain how the bytecode will be rewritten for each control structure found. After that, major problems are solved and we have a working prototype.
  • このエントリーをはてなブックマークに追加