If you’ve ever wanted to simplify your command-line scripting or use Python alongside command-line applications—or any applications for that matter—then the Python 0 module can help. From running shell commands and command-line applications to launching GUI applications, the Python 0 module can help. Show
By the end of this tutorial, you’ll be able to:
In this tutorial, you’ll get a high-level mental model for understanding processes, subprocesses, and Python before getting stuck into the 0 module and experimenting with an example. After that, you’ll start exploring the shell and learn how you can leverage Python’s 0 with Windows and UNIX-based shells and systems. Specifically, you’ll cover communication with processes, pipes and error handling.Note: 0 isn’t a GUI automation module or a way to achieve concurrency. For GUI automation, you might want to look at PyAutoGUI. For concurrency, take a look at this tutorial’s section on modules related to 0.Once you have the basics down, you’ll be exploring some practical ideas for how to leverage Python’s 0. You’ll also dip your toes into advanced usage of Python’s 0 by experimenting with the underlying 3 constructor.Source Code: Click here to download the free source code that you’ll use to get acquainted with the Python 0 module.Processes and SubprocessesFirst off, you might be wondering why there’s a 5 in the Python 0 module name. And what exactly is a process, anyway? In this section, you’ll answer these questions. You’ll come away with a high-level mental model for thinking about processes. If you’re already familiar with processes, then you might want to skip directly to basic usage of the Python 0 module.Remove adsProcesses and the Operating SystemWhenever you use a computer, you’ll always be interacting with programs. A process is the operating system’s abstraction of a running program. So, using a computer always involve processes. Start menus, app bars, command-line interpreters, text editors, browsers, and more—every application comprises one or more processes. A typical operating system will report hundreds or even thousands of running processes, which you’ll get to explore shortly. However, central processing units (CPUs) typically only have a handful of cores, which means that they can only run a handful of instructions simultaneously. So, you may wonder how thousands of processes can appear to run at the same time. In short, the operating system is a marvelous multitasker—as it has to be. The CPU is the brain of a computer, but it operates at the nanosecond timescale. Most other components of a computer are far slower than the CPU. For instance, a magnetic hard disk read takes thousands of times longer than a typical CPU operation. If a process needs to write something to the hard drive, or wait for a response from a remote server, then the CPU would sit idle most of the time. Multitasking keeps the CPU busy. Part of what makes the operating system so great at multitasking is that it’s fantastically organized too. The operating system keeps track of processes in a process table or process control block. In this table, you’ll find the process’s file handles, security context, references to its address spaces, and more. The process table allows the operating system to abandon a particular process at will, because it has all the information it needs to come back and continue with the process at a later time. A process may be interrupted many thousands of times during execution, but the operating system always finds the exact point where it left off upon returning. An operating system doesn’t boot up with thousands of processes, though. Many of the processes you’re familiar with are started by you. In the next section, you’ll look into the lifetime of a process. Process LifetimeThink of how you might start a Python application from the command line. This is an instance of your command-line process starting a Python process: The process that starts another process is referred to as the parent, and the new process is referred to as the child. The parent and child processes run mostly independently. Sometimes the child inherits specific resources or contexts from the parent. As you learned in Processes and the Operating System, information about processes is kept in a table. Each process keeps track of its parents, which allows the process hierarchy to be represented as a tree. You’ll be exploring your system’s process tree in the next section. Note: The precise mechanism for creating processes differs depending on the operating system. For a brief overview, the Wikipedia article on process management has a short section on process creation. For more details about the Windows mechanism, check out the win32 API documentation page on creating processes On UNIX-based systems, processes are typically created by using 8 to copy the current process and then replacing the child process with one of the 9 family of functions.The parent-child relationship between a process and its subprocess isn’t always the same. Sometimes the two processes will share specific resources, like inputs and outputs, but sometimes they won’t. Sometimes child processes live longer than the parent. A child outliving the parent can lead to orphaned or zombie processes, though more discussion about those is outside the scope of this tutorial. When a process has finished running, it’ll usually end. Every process, on exit, should return an integer. This integer is referred to as the return code or exit status. Zero is synonymous with success, while any other value is considered a failure. Different integers can be used to indicate the reason why a process has failed. In the same way that you can return a value from a function in Python, the operating system expects an integer return value from a process once it exits. This is why the canonical C 0 function usually returns an integer:
This example shows a minimal amount of C code necessary for the file to compile with 1 without any warnings. It has a 0 function that returns an integer. When this program runs, the operating system will interpret its execution as successful since it returns zero.So, what processes are running on your system right now? In the next section, you’ll explore some of the tools that you can use to take a peek at your system’s process tree. Being able to see what processes are running and how they’re structured will come in handy when visualizing how the 0 module works.Remove adsActive Processes on Your SystemYou may be curious to see what processes are running on your system right now. To do that, you can use platform-specific utilities to track them:
There are many tools available for Windows, but one which is easy to get set up, is fast, and will show you the process tree without much effort is Process Hacker. You can install Process Hacker by going to the downloads page or with Chocolatey:
Open the application, and you should immediately see the process tree. One of the native commands that you can use with PowerShell is 4, which lists the active processes on the command line. 5 is a command prompt utility that does the same.The official Microsoft version of Process Hacker is part of the Sysinternals utilities, namely Process Monitor and Process Explorer. You also get PsList, which is a command-line utility similar to 6 on UNIX. You can install Sysinternals by going to the downloads page or by using Chocolatey:
You can also use the more basic, but classic, Task Manager—accessible by pressing Win+X and selecting the Task Manager. For UNIX-based systems, there are many command-line utilities to choose from:
On macOS, you also have the Activity Monitor application in your utilities. In the View menu, if you select All Processes, Hierarchically, you should be able to see your process tree. You can also explore the Python psutil library, which allows you to retrieve running process information on both Windows and UNIX-based systems. One universal attribute of process tracking across systems is that each process has a process identification number, or PID, which is a unique integer to identify the process within the context of the operating system. You’ll see this number on most of the utilities listed above. Along with the PID, it’s typical to see the resource usage, such as CPU percentage and amount of RAM that a particular process is using. This is the information that you look for if a program is hogging all your resources. The resource utilization of processes can be useful for developing or debugging scripts that use the 0 module, even though you don’t need the PID, or any information about what resources processes are using in the code itself. While playing with the examples that are coming up, consider leaving a representation of the process tree open to see the new processes pop up.You now have a bird’s-eye view of processes. You’ll deepen your mental model throughout the tutorial, but now it’s time to see how to start your own processes with the Python 0 module.Overview of the Python $ python timer.py 5 0 ModuleThe Python 0 module is for launching child processes. These processes can be anything from GUI applications to the shell. The parent-child relationship of processes is where the sub in the 0 name comes from. When you use 0, Python is the parent that creates a new child process. What that new child process is, is up to you.Python 0 was originally proposed and accepted for Python 2.4 as an alternative to using the 3 module. Some documented changes have happened as late as 3.8. The examples in this article were tested with Python 3.10.4, but you only need 3.8+ to follow along with this tutorial.Most of your interaction with the Python 0 module will be via the 5 function. This blocking function will start a process and wait until the new process exits before moving on.The documentation recommends using 5 for all cases that it can handle. For edge cases where you need more control, the 7 class can be used. 7 is the underlying class for the whole 0 module. All functions in the 0 module are convenience wrappers around the 3 constructor and its instance methods. Near the end of this tutorial, you’ll dive into the 7 class.Note: If you’re trying to decide whether you need 0 or not, check out the section on deciding whether you need 0 for your task.You may come across other functions like 05, 06, and 07, but these belong to the older 0 API from Python 3.5 and earlier. Everything these three functions do can be replicated with the newer 5 function. The older API is mainly still there for backwards compatibility, and you won’t cover it in this tutorial.There’s also a fair amount of redundancy in the 0 module, meaning that there are various ways to achieve the same end goal. You won’t be exploring all variations in this tutorial. What you will find, though, are robust techniques that should keep you on the right path.Basic Usage of the Python $ python timer.py 5 0 ModuleIn this section, you’ll take a look at some of the most basic examples demonstrating the usage of the 0 module. You’ll start by exploring a bare-bones command-line timer program with the 5 function.If you want to follow along with the examples, then create a new folder. All the examples and programs can be saved in this folder. Navigate to this newly created folder on the command line in preparation for the examples coming up. All the code in this tutorial is standard library Python—with no external dependencies required—so a virtual environment isn’t necessary. Remove adsThe Timer ExampleTo come to grips with the Python 0 module, you’ll want a bare-bones program to run and experiment with. For this, you’ll use a program written in Python:
The timer program uses 15 to accept an integer as an argument. The integer represents the number of seconds that the timer should wait until exiting, which the program uses 16 to achieve. It’ll play a small animation representing each passing second until it exits:It’s not much, but the key is that it serves as a cross-platform process that runs for a few seconds and which you can easily tinker with. You’ll be calling it with 0 as if it were a separate executable.Note: Calling Python programs with the Python 0 module doesn’t make much sense—there’s usually no need for other Python modules to be in separate processes since you can just import them.The main reason you’ll be using Python programs for most of the examples in this tutorial is that they’re cross-platform, and you most likely already have Python installed! You may be tempted to think that starting a new process could be a neat way to achieve concurrency, but that’s not the intended use case for the 0 module. Maybe what you need are other Python modules dedicated to concurrency, covered in a later section.The 0 module is mainly for calling programs other than Python. But, as you can see, you can call Python too if you want! For more discussion on the use cases of 0, check out the section where this is discussed in more depth, or one of the later examples.Okay, ready to get stuck in! Once you have the 22 program ready, open a Python interactive session and call the timer with 0:>>>
With this code, you should’ve seen the animation playing right in the REPL. You imported 0 and then called the 5 function with a list of strings as the one and only argument. This is the 26 parameter of the 5 function.On executing 5, the timer process starts, and you can see its output in real time. Once it’s done, it returns an instance of the 29 class.On the command line, you might be used to starting a program with a single string:
However, with 5 you need to pass the command as a sequence, as shown in the 5 example. Each item in the sequence represents a token which is used for a system call to start a new process.Note: Calling 5 isn’t the same as calling programs on the command line. The 5 function makes a system call, foregoing the need for a shell. You’ll cover interaction with the shell in a later section.Shells typically do their own tokenization, which is why you just write the commands as one long string on the command line. With the Python 0 module, though, you have to break up the command into tokens manually. For instance, executable names, flags, and arguments will each be one token.Note: You can use the 35 module to help you out if you need, just bear in mind that it’s designed for POSIX compliant systems and may not work well in Windows environments:>>>
The 36 function divides a typical command into the different tokens needed. The 35 module can come in handy when it may be not obvious how to divide up more complex commands that have special characters, like spaces:>>>
You’ll note that the message, which contains spaces, is preserved as a single token, and the extra quotation marks are no longer needed. The extra quotation marks on the shell serve to group the token together, but since 0 uses sequences, it’s always unambiguous which parts should be interpreted as one token.Now that you’re familiar with some of the very basics of starting new processes with the Python 0 module, coming up you’ll see that you can run any kind of process, not just Python or text-based programs.The Use of $ python timer.py 5 0 to Run Any AppWith 0, you aren’t limited to text-based applications like the shell. You can call any application that you can with the Start menu or app bar, as long as you know the precise name or path of the program that you want to run:
>>>
>>>
Depending on your Linux distribution, you may have a different text editor, such as 42, 43, 44, or 45.>>> 0These commands should open up a text editor window. Usually 29 won’t get returned until you close the editor window. Yet in the case of macOS, since you need to run the launcher process 47 to launch TextEdit, the 29 gets returned straight away.Launcher processes are in charge of launching a specific process and then ending. Sometimes programs, such as web browsers, have them built in. The mechanics of launcher processes is out of the scope of this tutorial, but suffice to say that they’re able to manipulate the operating system’s process tree to reassign parent-child relationships. Note: There are many problems that you might initially reach for 0 to solve, but then you’ll find a specific module or library that solves it for you. This tends to be a theme with 0 since it is quite a low-level utility.An example of something that you might want to do with 0 is to open a web browser to a specific page. However, for that, it’s probably best to use the Python module 52. The 52 module uses 0 under the hood but handles all the finicky cross-platform and browser differences that you might encounter.Then again, 0 can be a remarkably useful tool to get something done quickly. If you don’t need a full-fledged library, then 0 can be your Swiss Army knife. It all depends on your use case. More discussion on this topic will come later.You’ve successfully started new processes using Python! That’s 0 at its most basic. Next up, you’ll take a closer look at the 29 object that’s returned from 5.Remove adsThe PS> choco install processhacker 29 ObjectWhen you use 5, the return value is an instance of the 29 class. As the name suggests, 5 returns the object only once the child process has ended. It has various attributes that can be helpful, such as the 26 that were used for the process and the 65.To see this clearly, you can assign the result of 5 to a variable, and then access its attributes such as 67:>>> 1The process has a return code that indicates failure, but it doesn’t raise an exception. Typically, when a 0 process fails, you’ll always want an exception to be raised, which you can do by passing in a 69 argument:>>> 2There are various ways to deal with failures, some of which will be covered in the next section. The important point to note for now is that 5 won’t necessarily raise an exception if the process fails unless you’ve passed in a 69 argument.The 29 also has a few attributes relating to input/output (I/O), which you’ll cover in more detail in the communicating with processes section. Before communicating with processes, though, you’ll learn how to handle errors when coding with 0.$ python timer.py 5 0 ExceptionsAs you saw earlier, even if a process exits with a return code that represents failure, Python won’t raise an exception. For most use cases of the 0 module, this isn’t ideal. If a process fails, you’ll usually want to handle it somehow, not just carry on.A lot of 0 use cases involve short personal scripts that you might not spend much time on, or at least shouldn’t spend much time on. If you’re tinkering with a script like this, then you’ll want 0 to fail early and loudly.PS> choco install processhacker 78 for Non-Zero Exit CodeIf a process returns an exit code that isn’t zero, you should interpret that as a failed process. Contrary to what you might expect, the Python 0 module does not automatically raise an exception on a non-zero exit code. A failing process is typically not something you want your program to pass over silently, so you can pass a 69 argument to 5 to raise an exception:>>> 2The 78 is raised as soon as the subprocess runs into a non-zero return code. If you’re developing a short personal script, then perhaps this is good enough for you. If you want to handle errors more gracefully, then read on to the section on exception handling.One thing to bear in mind is that the 78 does not apply to processes that may hang and block your execution indefinitely. To guard against that, you’d want to take advantage of the 84 parameter.PS> choco install processhacker 85 for Processes That Take Too LongSometimes processes aren’t well behaved, and they might take too long or just hang indefinitely. To handle those situations, it’s always a good idea to use the 84 parameter of the 5 function.Passing a 88 argument to 5 will cause the function to shut down the process and raise a 85 error after one second:>>> 4In this example, the first dot of the timer animation was output, but the subprocess was shut down before being able to complete. The other type of error that might happen is if the program doesn’t exist on that particular system, which raises one final type of error. Remove adsPS> choco install processhacker 91 for Programs That Don’t ExistThe final type of exception you’ll be looking at is the 91, which is raised if you try and call a program that doesn’t exist on the target system:>>> 5This type of error is raised no matter what, so you don’t need to pass in any arguments for the 91.Those are the main exceptions that you’ll run into when using the Python 0 module. For many use cases, knowing the exceptions and making sure that you use 84 and 96 arguments will be enough. That’s because if the subprocess fails, then that usually means that your script has failed.However, if you have a more complex program, then you may want to handle errors more gracefully. For instance, you may need to call many processes over a long period of time. For this, you can use the 97 … 98 construct.An Example of Exception HandlingHere’s a code snippet that shows the main exceptions that you’ll want to handle when using 0: 6This snippet shows you an example of how you might handle the three main exceptions raised by the 0 module.Now that you’ve used 0 in its basic form and handled some exceptions, it’s time to get familiar with what it takes to interact with the shell.Introduction to the Shell and Text-Based Programs With $ python timer.py 5 0Some of the most popular use cases of the 0 module are to interact with text-based programs, typically available on the shell. That’s why in this section, you’ll start to explore all the moving parts involved when interacting with text-based programs, and perhaps question if you need the shell at all!The shell is typically synonymous with the command-line interface or CLI, but this terminology isn’t entirely accurate. There are actually two separate processes that make up the typical command-line experience:
When on the command line, it’s common to think that you’re interacting directly with the shell, but you’re really interacting with the interface. The interface takes care of sending your commands to the shell and displaying the shell’s output back to you. With this important distinction in mind, it’s time to turn your attention to what 5 is actually doing. It’s common to think that calling 5 is somehow the same as typing a command in a terminal interface, but there are important differences.While all new process are created with the same system calls, the context from which the system call is made is different. The 5 function can make a system call directly and doesn’t need to go through the shell to do so:In fact, many programs that are thought of as shell programs, such as Git, are really just text-based programs that don’t need a shell to run. This is especially true of UNIX environments, where all of the familiar utilities like 3, 08, 09, and 10 are actually separate executables that can be called directly:>>> 7There are some tools that are specific to shells, though. Finding tools embedded within the shell is far more common on Windows shells like PowerShell, where commands like 3 are part of the shell itself and not separate executables like they are in a UNIX environment:>>> 8In PowerShell, 3 is the default alias for 13, but calling that won’t work either because 13 isn’t a separate executable—it’s part of PowerShell itself.The fact that many text-based programs can operate independently from the shell may make you wonder if you can cut out the middle process—namely, the shell—and use 0 directly with the text-based programs typically associated with the shell.Remove adsUse Cases for the Shell and $ python timer.py 5 0There are a few common reasons why you might want to call the shell with the 17 subprocess module:
This isn’t an exhaustive list! You might use the shell to wrap programs or to do some text processing. However, the syntax can be very cryptic when compared to Python. With Python, text processing workflows are easier to write, easier to maintain, generally more performant, and cross-platform to boot. So it’s well worth considering going without the shell. What often happens, though, is that you just don’t have the time or it’s not worth the effort to reimplement existing shell scripts in Python. In those cases, using 0 for some sloppy Python isn’t a bad thing!Common reasons for using 0 itself are similar in nature to using the shell with 0:
Note: A black box could be a program that can be freely used but whose source code isn’t available, so there’s no way to know exactly what it does and no way to modify its internals. Similarly, a white box could be a program whose source code is available but can’t be changed. It could also be a program whose source code you could change, but its complexity means that it would take you a long time to get your head around it to be able to change it. In these cases, you can use 0 to wrap your boxes of varying opacity, bypassing any need to change or reimplement things in Python.Often you’ll find that for 0 use cases, there will be a dedicated library for that task. Later in the tutorial, you’ll examine a script that creates a Python project, complete with a virtual environment and a fully initialized Git repository. However, the Cookiecutter and Copier libraries already exist for that purpose.Even though specific libraries might be able to do your task, it may still be worth doing things with 0. For one, it might be much faster for you to execute what you already know how to do, rather than learning a new library.Additionally, if you’re sharing this script with friends or colleagues, it’s convenient if your script is pure Python without any other dependencies, especially if your script needs to go on minimal environments like servers or embedded systems. However, if you’re using 0 instead of 25 to read and write a few files with Bash, you might want to consider learning how to read and write with Python. Learning how to read and write files doesn’t take long, and it’ll definitely be worth it for such a common task.With that out of the way, it’s time to get familiar with the shell environments on both Windows and UNIX-based systems. Basic Usage of $ python timer.py 5 0 With UNIX-Based ShellsTo run a shell command using 5, the 26 should contain the shell that you want to use, the flag to indicate that you want it to run a specific command, and the command that you’re passing in:>>> 9Here a common shell command is demonstrated. It uses 3 piped into 09 to filter some of the entries. The shell is handy for this kind of operation because you can take advantage of the pipe operator ( 31). You’ll cover pipes in more detail later.You can replace 32 with the shell of your choice. The 33 flag stands for command, but may be different depending on the shell that you’re using. This is almost the exact equivalent of what happens when you add the 34 argument:>>> 0The 34 argument uses 36 behind the scenes, so it’s almost the equivalent of the previous example.Note: On UNIX-based systems, the 37 shell was traditionally the Bourne shell. That said, the Bourne shell is now quite old, so many operating systems use 37 as a link to Bash or Dash.This can often be different from the shell used with the terminal interface that you interact with. For instance, since macOS Catalina, the default shell that you’ll find on the command-line app has changed from Bash to Zsh, yet 37 often still points to Bash. Likewise, on Ubuntu, 37 points to Dash, but the default that you typically interact with on the command-line application is still Bash.So, calling 37 on your system may result in a different shell than what is found in this tutorial. Nevertheless, the examples should all still work.You’ll note that the token after 42 should be one single token, with all the spaces included. Here you’re giving control to the shell to parse the command. If you were to include more tokens, this would be interpreted as more options to pass to the shell executable, not as additional commands to run inside the shell.Remove adsBasic Usage of $ python timer.py 5 0 With Windows ShellsIn this section, you’ll cover basic use of the shell with 0 in a Windows environment.To run a shell command using 5, the 26 should contain the shell that you want to use, the flag to indicate that you want it to run a specific command, and the command that you’re passing in:>>> 1Note that 47 and 48 both work. If you don’t have PowerShell Core, then you can call 49 or 50.You’ll note that the token after 51 should be one single token, with all the spaces included. Here you’re giving control to the shell to parse the command. If you were to include more tokens, this would be interpreted as more options to pass to the shell executable, not as additional commands to run inside the shell.If you need the Command Prompt, then the executable is 52 or 53, and the flag to indicate that the following token is a command is 54:>>> 2This last example is the exact equivalent of calling 5 with 34. Said in another way, using the 34 argument is like prepending 58 and 59 to your argument list.Note: Windows’ evolution has been very different from that of UNIX-based systems. The most widely known shell is the Windows Command Prompt which is by now a legacy shell. The Command Prompt was made to emulate the pre-Windows MS-DOS environment. Many shell scripts, or batch 60 scripts, were written for this environment which are still in use today.The 5 function with the 62 parameter will almost always end up using the Command Prompt. The 0 module uses the Windows 64 environment variable, which in almost all cases will point to 53, the Command Prompt. By now, there are so many programs that equate 64 to 53 that changing it would cause much breakage in unexpected places! So, changing 64 is generally not advised.At this point, you should know about an important security concern that you’ll want to be aware of if you have user-facing elements in your Python program, regardless of the operating system. It’s a vulnerability that’s not confined to 0. Rather, it can be exploited in many different areas.A Security WarningIf at any point you plan to get user input and somehow translate that to a call to 0, then you have to be very careful of injection attacks. That is, take into account potential malicious actors. There are many ways to cause havoc if you just let people run code on your machine.To use a very simplistic example, where you take user input and send it, unfiltered, to subprocess to run on the shell:
3 4You can imagine the intended use case is to wrap 3 and add something to it. So the expected user behavior is to provide a path like 72. However, if a malicious actor realized what was happening, they could execute almost any code they wanted. Take the following, for instance, but be careful with this:
73 74 Again, beware! These innocent-looking lines could try and delete everything on the system! In this case the malicious part is in quotes, so it won’t run, but if the quotes were not there, you’d be in trouble. The key part that does this is the call to 08 with the relevant flags to recursively delete all files, folders, and subfolders, and it’ll work to force the deletion through. It can run the 76 and potentially the 08 as entirely separate commands by adding semicolons, which act as command separators allowing what would usually be multiple lines of code to run on one line.Running these malicious commands would cause irreparable damage to the file system, and would require reinstalling the operating system. So, beware! Luckily, the operating system wouldn’t let you do this to some particularly important files. The 08 command would need to use 79 in UNIX-based systems, or be run as an administrator in Windows to be completely successful in its mayhem. The command would probably delete a lot of important stuff before stopping, though.So, make sure that if you’re dynamically building user inputs to feed into a 0 call, then you’re very careful! With that warning, coming up you’ll be covering using the outputs of commands and chaining commands together—in short, how to communicate with processes once they’ve started.Remove adsCommunication With ProcessesYou’ve used the 0 module to execute programs and send basic commands to the shell. But something important is still missing. For many tasks that you might want to use 0 for, you might want to dynamically send inputs or use the outputs in your Python code later.To communicate with your process, you first should understand a little bit about how processes communicate in general, and then you’ll take a look at two examples to come to grips with the concepts. The Standard I/O StreamsA stream at its most basic represents a sequence of elements that aren’t available all at once. When you read characters and lines from a file, you’re working with a stream in the form of a file object, which at its most basic is a file descriptor. File descriptors are often used for streams. So, it’s not uncommon to see the terms stream, file, and file-like used interchangeably. When processes are initialized, there are three special streams that a process makes use of. A process does the following:
These are the standard streams—a cross-platform pattern for process communication. Sometimes the child process inherits these streams from the parent. This is what’s happening when you use 86 in the REPL and are able to see the output of the command. The 84 of the Python interpreter is inherited by the subprocess.When you’re in a REPL environment, you’re looking at a command-line interface process, complete with the three standard I/O streams. The interface has a shell process as a child process, which itself has a Python REPL as a child. In this situation, unless you specify otherwise, 83 comes from the keyboard, while 84 and 85 are displayed on-screen. The interface, the shell, and the REPL share the streams:You can think of the standard I/O streams as byte dispensers. The subprocess fills up 84 and 85, and you fill up 83. Then you read the bytes in 84 and 85, and the subprocess reads from 83.As with a dispenser, you can stock 83 before it gets linked up to a child process. The child process will then read from 83 as and when it needs to. Once a process has read from a stream, though, the bytes are dispensed. You can’t go back and read them again:These three streams, or files, are the basis for communicating with your process. In the next section, you’ll start to see this in action by getting the output of a magic number generator program. The Magic Number Generator ExampleOften, when using the 0 module, you’ll want to use the output for something and not just display the output as you have been doing so far. In this section, you’ll use a magic number generator that outputs, well, a magic number.Imagine that the magic number generator is some obscure program, a black box, inherited across generations of sysadmins at your job. It outputs a magic number that you need for your secret calculations. You’ll read from the 84 of 0 and use it in your wrapper Python program: 5Okay, not really so magical. That said, it’s not the magic number generator that you’re interested in—it’s interacting with a hypothetical black box with 0 that’s interesting. To grab the number generator’s output to use later, you can pass in a 03 argument to 5:>>> 6Passing a 05 argument of 06 to 5 makes the output of the process available at the 08 attribute of the completed process object. You’ll note that it’s returned as a bytes object, so you need to be mindful of encodings when reading it.Also note that the 08 attribute of the 29 is no longer a stream. The stream has been read, and it’s stored as a bytes object in the 08 attribute.With the output available, you can use more than one subprocess to grab values and operate on them in your code: >>> 7In this example, you start two magic number processes that fetch two magic numbers and then add them together. For now, you rely on the automatic decoding of the bytes object by the 12 constructor. In the next section, though, you’ll learn how to decode and encode explicitly.Remove adsThe Decoding of Standard StreamsProcesses communicate in bytes, and you have a few different ways to deal with encoding and decoding these bytes. Beneath the surface, 0 has a few ways of getting into text mode.Text mode means that 0 will try to take care of encoding itself. To do that, it needs to know what character encoding to use. Most of the options for doing this in 0 will try to use the default encoding. However, you generally want to be explicit about what encoding to use to prevent a bug that would be hard to find in the future.You can pass a 16 argument for Python to take care of encodings using the default encoding. But, as mentioned, it’s always safer to specify the encodings explicitly using the 17 argument, as not all systems work with the nearly universal UTF-8:>>> 8If in text mode, the 08 attribute on a 29 is now a string and not a bytes object.You can also decode the bytes returned by calling the 20 method on the 84 attribute directly, without requiring text mode at all:>>> 9There are other ways to put 5 into text mode. You can also set a 06 value for 24 or 25, which will also put 5 into text mode. This may seem redundant, but much of this is kept for backwards compatibility, seeing as the 0 module has changed over the years.Now that you know how to read and decode the output of a process, it’s time to take a look at writing to the input of a process. Reaction Game ExampleIn this section, you’ll use 0 to interact with a command-line game. It’s a basic program that’s designed to test a human’s reaction time. With your knowledge of standard I/O streams, though, you’ll be able to hack it! The source code of the game makes use of the 29 and 30 module: 0The program starts, asks for the user to press enter, and then after a random amount of time will ask the user to press enter again. It measures from the time the message appears to the time the user presses enter, or at least that’s what the game developer thinks: The 31 function will read from 83 until it reaches a newline, which means an Enter keystroke in this context. It returns everything it consumed from 83 except the newline. With that knowledge, you can use 0 to interact with this game:>>> 1A reaction time of 0 milliseconds! Not bad! Considering the average human reaction time is around 270 milliseconds, your program is definitely superhuman. Note that the game rounds its output, so 0 milliseconds doesn’t mean it’s instantaneous. The 35 argument passed to 5 is a string consisting of two newlines. The 17 parameter is set to 38, which puts 5 into text mode. This sets up the process for it to receive the input you that give it.Before the program starts, 83 is stocked, waiting for the program to consume the newlines it contains. One newline is consumed to start the game, and the next newline is consumed to react to 41.Now that you know what’s happening—namely that 83 can be stocked, as it were—you can hack the program yourself without 0. If you start the game and then press Enter a few times, that’ll stock up 83 with a few newlines that the program will automatically consume once it gets to the 31 line. So your reaction time is really only the time it takes for the reaction game to execute 46 and consume an input:The game developer gets wise to this, though, and vows to release another version, which will guard against this exploit. In the meantime, you’ll peek a bit further under the hood of 0 and learn about how it wires up the standard I/O streams.Remove adsPipes and the ShellTo really understand subprocesses and the redirection of streams, you really need to understand pipes and what they are. This is especially true if you want to wire up two processes together, feeding one 84 into another process’s 83, for instance. In this section, you’ll be coming to grips with pipes and how to use them with the 0 module.Introduction to PipesA pipe, or pipeline, is a special stream that, instead of having one file handle as most files do, has two. One handle is read-only, and the other is write-only. The name is very descriptive—a pipe serves to pipe a byte stream from one process to another. It’s also buffered, so a process can write to it, and it’ll hold onto those bytes until it’s read, like a dispenser. You may be used to seeing pipes on the command line, as you did in the section on shells: 2This command tells the shell to create an 3 process to list all the files in 52. The pipe operator ( 31) tells the shell to create a pipe from the 84 of the 3 process and feed it into the 83 of the 09 process. The 09 process filters out all the lines that don’t contain the string 59.Windows doesn’t have 09, but a rough equivalent of the same command would be as follows: 3However, on Windows PowerShell, things work very differently. As you learned in the Windows shell section of this tutorial, the different commands are not separate executables. Therefore, PowerShell is internally redirecting the output of one command into another without starting new processes. Note: If you don’t have access to a UNIX-based operating system but have Windows 10 or above, then you actually do have access to a UNIX-based operating system! Check out Windows Subsystem for Linux, which will give you access to a fully featured Linux shell. You can use pipes for different processes on PowerShell, though getting into the intricacies of which ones is outside the scope of this tutorial. For more information on PowerShell pipes, check out the documentation. So, for the rest of the pipe examples, only UNIX-based examples will be used, as the basic mechanism is the same for both systems. They’re not nearly as common on Windows, anyway. If you want to let the shell take care of piping processes into one another, then you can just pass the whole string as a command into 0:>>> 4This way, you can let your chosen shell take care of piping one process into another, instead of trying to reimplement things in Python. This is a perfectly valid choice in certain situations. Later in the tutorial, you’ll also come to see that you can’t pipe processes directly with 5. For that, you’ll need the more complicated 3. Actual piping is demonstrated in Connecting Two Porcesses Together With Pipes, near the end of the tutorial.Whether you mean to pipe one process into another with the 0 module or not, the 0 module makes extensive use of pipes behind the scenes.The Pipes of $ python timer.py 5 0The Python 0 module uses pipes extensively to interact with the processes that it starts. In a previous example, you used the 05 parameter to be able to access 84:>>> 6 03 is equivalent to explicitly setting the 84 and 85 parameters to the 73 constant:>>> 6The 74 constant is nothing special. It’s just a number that indicates to 0 that a pipe should be created. The function then creates a pipe to link up to the 84 of the subprocess, which the function then reads into the 29 object’s 84 attribute. By the time it’s a 29, it’s no longer a pipe, but a bytes object that can be accessed multiple times.Note: Pipe buffers have a limited capacity. Depending on the system you are running on, you may easily run into that limit if you plan on holding large quantities of data in the buffer. To work around this limit, you can use normal files. You can also pass a file object to any of the standard stream parameters: >>> 7You can’t pass a bytes object or a string directly to the 83 argument, though. It needs to be something file-like.Note that the 81 that gets returned first is from the call to 82 which returns the new stream position, which in this case is the start of the stream.The 35 parameter is similar to the 05 parameter in that it’s a shortcut. Using the 35 parameter will create a buffer to store the contents of 35, and then link the file up to the new process to serve as its 83.To actually link up two processes with a pipe from within 0 is something that you can’t do with 5. Instead, you can delegate the plumbing to the shell, as you did earlier in the Introduction to the Shell and Text Based Programs with 0 section.If you needed to link up different processes without delegating any of the work to the shell, then you could do that with the underlying 3 constructor. You’ll cover 3 in a later section. In the next section, though, you’ll be simulating a pipe with 5 because in most cases, it’s not vital for processes to be linked up directly.Pipe Simulation With >>> subprocess.run(["gedit"]) CompletedProcess(args=['gedit'], returncode=0) 5Though you can’t actually link up two processes together with a pipe by using the 5 function, at least not without delegating it to the shell, you can simulate piping by judicious use of the 84 attribute.If you’re on a UNIX-based system where almost all typical shell commands are separate executables, then you can just set the 35 of the second process to the 08 attribute of the first 29:>>> 8Here the 08 attribute of the 29 object of 3 is set to the 35 of the 04. It’s important that it’s set to 35 rather than 83. This is because the 08 attribute isn’t a file-like object. It’s a bytes object, so it can’t be used as an argument to 83.As an alternative, you can operate directly with files too, setting them to the standard stream parameters. When using files, you set the file object as the argument to 83, instead of using the 35 parameter:>>> 9As you learned in the previous section, for Windows PowerShell, doing something like this doesn’t make a whole lot of sense because most of the time, these utilities are part of PowerShell itself. Because you aren’t dealing with separate executables, piping becomes less of a necessity. However, the pattern for piping is still the same if something like this needs to be done. With most of the tools out the way, it’s now time to think about some practical applications for 0.Practical IdeasWhen you have an issue that you want to solve with Python, sometimes the 0 module is the easiest way to go, even though it may not be the most correct.Using 0 is often tricky to get working across different platforms, and it has inherent dangers. But even though it may involve some sloppy Python, using 0 can be a very quick and efficient way to solve a problem.As mentioned, for most tasks you can imagine doing with 0, there’s usually a library out there that’s dedicated to that specific task. The library will almost certainly use 0, and the developers will have worked hard to make the code reliable and to cover all the corner cases that can make using 0 difficult.So, even though dedicated libraries exist, it can often be simpler to just use 0, especially if you’re in an environment where you need to limit your dependencies.In the following sections, you’ll be exploring a couple of practical ideas. Creating a New Project: An ExampleSay you often need to create new local projects, each complete with a virtual environment and initialized as a Git repository. You could reach for the Cookiecutter library, which is dedicated to that task, and that wouldn’t be a bad idea. However, using Cookiecutter would mean learning Cookiecutter. Imagine you didn’t have much time, and your environment was extremely minimal anyway—all you could really count on was Git and Python. In these cases, 0 can quickly set up your project for you: 0This is a command-line tool that you can call to start a project. It’ll take care of creating a 20 file and a 21 file, and then it’ll run a few commands to create a virtual environment, initialize a git repository, and perform your first commit. It’s even cross-platform, opting to use 25 to create the files and folders, which abstracts away the operating system differences.Could this be done with Cookiecutter? Could you use GitPython for the 23 part? Could you use the 24 module to create the virtual environment? Yes to all. But if you just need something quick and dirty, using commands you already know, then just using 0 can be a great option.Changing Extended AttributesIf you use Dropbox, you may not know that there’s a way to ignore files when syncing. For example, you can keep virtual environments in your project folder and use Dropbox to sync the code, but keep the virtual environment local. That said, it’s not as easy as adding a 26 file. Rather, it involves adding special attributes to files, which can be done from the command line. These attributes are different between UNIX-like systems and Windows:
1 2 3There are some UNIX-based projects, like dropboxignore, that use shell scripts to make it easier to ignore files and folders. The code is relatively complex, and it won’t work on Windows. With the 0 module, you can wrap the different shell commands quite easily to come up with your own utility: 4This is a simplified snippet from the author’s dotDropboxIgnore repository. The 28 function detects the operating system with the 29 module and returns an object that’s an abstraction around the system-specific shell. The code hasn’t implemented the behavior on macOS, so it raises a 30 if it detects it’s running on macOS.The shell object allows you to call an 31 method with a list of 25 33 objects to set Dropbox to ignore those files.On the 34 class, the constructor tests to see if PowerShell Core is available, and if not, will fall back to the older Windows PowerShell, which is installed by default on Windows 10.In the next section, you’ll review some of the other modules that might be interesting to keep in mind when deciding whether to use 0.Python Modules Associated With $ python timer.py 5 0When deciding whether a certain task is a good fit for 0, there are some associated modules that you may want to be aware of.Before 0 existed, you could use 39 to run commands. However, as with many things that 3 was used for before, standard library modules have come to replace 3, so it’s mostly used internally. There are hardly any use cases for using 3 yourself.There’s an official documentation page where you can examine some of the old ways to accomplish tasks with 3 and learn how you might do the same with 0.It might be tempting to think that 0 can be used for concurrency, and in simple cases, it can be. But, in line with the sloppy Python philosophy, it’s probably only going to be to hack something together quickly. If you want something more robust, then you’ll probably want to start looking at the 46 module.Depending on the task that you’re attempting, you may be able to accomplish it with the 47 or 48 modules. If everything is written in Python, then these modules are likely your best bet.The 47 module has a high-level API to create and manage subprocesses too, so if you want more control over non-Python parallel processes, that might be one to check out.Now it’s time to get deep into 0 and explore the underlying 7 class and its constructor.The >>> subprocess.run(["gedit"]) CompletedProcess(args=['gedit'], returncode=0) 7 ClassAs mentioned, the underlying class for the whole 0 module is the 7 class and the 3 constructor. Each function in 0 calls the 3 constructor under the hood. Using the 3 constructor gives you lots of control over the newly started subprocesses.As a quick summary, 5 is basically the 3 class constructor, some setup, and then a call to the 61 method on the newly initialized 7 object. The 61 method is a blocking method that returns the 84 and 85 data once the process has ended.The name of 7 comes from a similar UNIX command that stands for pipe open. The command creates a pipe and then starts a new process that invokes the shell. The 0 module, though, doesn’t automatically invoke the shell.The 5 function is a blocking function, which means that interacting dynamically with a process isn’t possible with it. However, the 3 constructor starts a new process and continues, leaving the process running in parallel.The developer of the reaction game that you were hacking earlier has released a new version of their game, one in which you can’t cheat by loading 83 with newlines: 5Now the program will display a random character, and you need to press that exact character to have the game register your reaction time: What’s to be done? First, you’ll need to come to grips with using 3 with basic commands, and then you’ll find another way to exploit the reaction game.Using >>> import shlex >>> shlex.split("python timer.py 5") ['python', 'timer.py', '5'] >>> subprocess.run(shlex.split("python timer.py 5")) Starting timer of 5 seconds .....Done! CompletedProcess(args=['python', 'timer.py', '5'], returncode=0) 3Using the 3 constructor is very similar in appearance to using 5. If there’s an argument that you can pass to 5, then you’ll generally be able to pass it to 3. The fundamental difference is that it’s not a blocking call—rather than waiting until the process is finished, it’ll run the process in parallel. So you need to take this non-blocking nature into account if you want to read the new process’s output: 6This program calls the timer process in a context manager and assigns 84 to a pipe. Then it runs the 78 method on the 7 object and reads its 84.The 78 method is a basic method to check if a process is still running. If it is, then 78 returns 83. Otherwise, it’ll return the process’s exit code.Then the program uses 84 to try and read as many bytes as are available at 08.Note: If you put the 7 object into text mode and then called 87 on 08, the call to 87 would be blocking until it reached a newline. In this case, a newline would coincide with the end of the timer program. This behavior isn’t desired in this situation.To read as many bytes as are available at that time, disregarding newlines, you need to read with 84. It’s important to note that 84 is only available on byte streams, so you need to make sure to deal with encodings manually and not use text mode.The output of this program first prints 83 because the process hasn’t yet finished. The program then prints what is available in 84 so far, which is the starting message and the first character of the animation.After three seconds, the timer hasn’t finished, so you get 83 again, along with two more characters of the animation. After another three seconds, the process has ended, so 78 produces 81, and you get the final characters of the animation and 97: 7In this example, you’ve seen how the 3 constructor works very differently from 5. In most cases, you don’t need this kind of fine-grained control. That said, in the next sections, you’ll see how you can pipe one process into another, and how you can hack the new reaction game.Connecting Two Processes Together With PipesAs mentioned in a previous section, if you need to connect processes together with pipes, you need to use the 3 constructor. This is mainly because 5 is a blocking call, so by the time the next process starts, the first one has ended, meaning that you can’t directly link up to its 84.This procedure will only be demonstrated for UNIX systems, because piping in Windows is far less common, as mentioned in the simulating a pipe section: 8In this example, the two processes are started in parallel. They are joined with a common pipe, and the 03 loop takes care of reading the pipe at 84 to output the lines.A key point to note is that in contrast to 5, which returns a 29 object, the 3 constructor returns a 7 object. The standard stream attributes of a 29 point to bytes objects or strings, but the same attributes of a 7 object point to the actual streams. This allows you to communicate with processes as they’re running.Whether you really need to pipe processes into one another, though, is another matter. Ask yourself if there’s much to be lost by mediating the process with Python and using 5 exclusively. There are some situations in which you really need 7, though, such as hacking the new version of the reaction time game.Interacting Dynamically With a ProcessNow that you know you can use 3 to interact with a process dynamically as it runs, it’s time to turn that knowledge toward exploiting the reaction time game again: 9With this script, you’re taking complete control of the buffering of a process, which is why you pass in arguments such as 14 to the Python process and 15 to 16. These arguments are to ensure that no extra buffering is taking place.The script works by using a function that’ll search for one of a list of strings by grabbing one character at a time from the process’s 84. As each character comes through, the script will search for the string.Note: To make this work on both Windows and UNIX-based systems, two strings are searched for: either 18 or 19. The Windows-style carriage return along with the typical newline is required on Windows systems.After the script has found one of the target strings, which in this case is the sequence of characters before the target letter, it’ll then grab the next character and write that letter to the process’s 83 followed by a newline:At one millisecond, it’s not quite as good as the original hack, but it’s still very much superhuman. Well done! With all this fun aside, interacting with processes using 7 can be very tricky and is prone to errors. First, see if you can use 5 exclusively before resorting to the 3 constructor.If you really need to interact with processes at this level, the 47 module has a high-level API to create and manage subprocesses.The 47 subprocess functionality is intended for more complex uses of 0 where you may need to orchestrate various processes. This might be the case if you’re performing complex processing of many image, video, or audio files, for example. If you’re using 0 at this level, then you’re probably building a library.ConclusionYou’ve completed your journey into the Python 0 module. You should now be able to decide whether 0 is a good fit for your problem. You should also be able to decide whether you need to invoke the shell. Aside from that, you should be able to run subprocesses and interact with their inputs and outputs.You should also be able to start exploring the possibilities of process manipulation with the 3 constructor.Along the way, you’ve:
You’re now ready to bring a variety of executables into your Pythonic sphere of influence! Source Code: Click here to download the free source code that you’ll use to get acquainted with the Python 0 module.Mark as Completed 🐍 Python Tricks 💌 Get a short & sweet Python Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team. Send Me Python Tricks » About Ian Currie Ian is a Python nerd who uses it for everything from tinkering to helping people and companies manage their day-to-day and develop their businesses. » More about IanEach tutorial at Real Python is created by a team of developers so that it meets our high quality standards. The team members who worked on this tutorial are: Aldren Bartosz Geir Arne Jim Kate Master Real-World Python Skills With Unlimited Access to Real Python Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas: Level Up Your Python Skills » Master Real-World Python Skills Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas: Level Up Your Python Skills » What Do You Think? Rate this article: Tweet Share Share EmailWhat’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know. Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students. Get tips for asking good questions and get answers to common questions in our support portal. |