파이썬 subprocess - paisseon subprocess

If you’ve ever wanted to simplify your command-line scripting or use Python alongside command-line applications—or any applications for that matter—then the Python

$ python timer.py 5
0 module can help. From running shell commands and command-line applications to launching GUI applications, the Python
$ python timer.py 5
0 module can help.

Show

By the end of this tutorial, you’ll be able to:

  • Understand how the Python
    $ python timer.py 5
    
    0 module interacts with the operating system
  • Issue shell commands like
    $ python timer.py 5
    
    3 or
    $ python timer.py 5
    
    4
  • Feed input into a process and use its output.
  • Handle errors when using
    $ python timer.py 5
    
    0
  • Understand the use cases for
    $ python timer.py 5
    
    0 by considering practical examples

In this tutorial, you’ll get a high-level mental model for understanding processes, subprocesses, and Python before getting stuck into the

$ python timer.py 5
0 module and experimenting with an example. After that, you’ll start exploring the shell and learn how you can leverage Python’s
$ python timer.py 5
0 with Windows and UNIX-based shells and systems. Specifically, you’ll cover communication with processes, pipes and error handling.

Note:

$ python timer.py 5
0 isn’t a GUI automation module or a way to achieve concurrency. For GUI automation, you might want to look at PyAutoGUI. For concurrency, take a look at this tutorial’s section on modules related to
$ python timer.py 5
0.

Once you have the basics down, you’ll be exploring some practical ideas for how to leverage Python’s

$ python timer.py 5
0. You’ll also dip your toes into advanced usage of Python’s
$ python timer.py 5
0 by experimenting with the underlying
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor.

Source Code: Click here to download the free source code that you’ll use to get acquainted with the Python

$ python timer.py 5
0 module.

Processes and Subprocesses

First off, you might be wondering why there’s a

>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
5 in the Python
$ python timer.py 5
0 module name. And what exactly is a process, anyway? In this section, you’ll answer these questions. You’ll come away with a high-level mental model for thinking about processes. If you’re already familiar with processes, then you might want to skip directly to basic usage of the Python
$ python timer.py 5
0 module.

Remove ads

Processes and the Operating System

Whenever you use a computer, you’ll always be interacting with programs. A process is the operating system’s abstraction of a running program. So, using a computer always involve processes. Start menus, app bars, command-line interpreters, text editors, browsers, and more—every application comprises one or more processes.

A typical operating system will report hundreds or even thousands of running processes, which you’ll get to explore shortly. However, central processing units (CPUs) typically only have a handful of cores, which means that they can only run a handful of instructions simultaneously. So, you may wonder how thousands of processes can appear to run at the same time.

In short, the operating system is a marvelous multitasker—as it has to be. The CPU is the brain of a computer, but it operates at the nanosecond timescale. Most other components of a computer are far slower than the CPU. For instance, a magnetic hard disk read takes thousands of times longer than a typical CPU operation.

If a process needs to write something to the hard drive, or wait for a response from a remote server, then the CPU would sit idle most of the time. Multitasking keeps the CPU busy.

Part of what makes the operating system so great at multitasking is that it’s fantastically organized too. The operating system keeps track of processes in a process table or process control block. In this table, you’ll find the process’s file handles, security context, references to its address spaces, and more.

The process table allows the operating system to abandon a particular process at will, because it has all the information it needs to come back and continue with the process at a later time. A process may be interrupted many thousands of times during execution, but the operating system always finds the exact point where it left off upon returning.

An operating system doesn’t boot up with thousands of processes, though. Many of the processes you’re familiar with are started by you. In the next section, you’ll look into the lifetime of a process.

Process Lifetime

Think of how you might start a Python application from the command line. This is an instance of your command-line process starting a Python process:

파이썬 subprocess - paisseon subprocess

The process that starts another process is referred to as the parent, and the new process is referred to as the child. The parent and child processes run mostly independently. Sometimes the child inherits specific resources or contexts from the parent.

As you learned in Processes and the Operating System, information about processes is kept in a table. Each process keeps track of its parents, which allows the process hierarchy to be represented as a tree. You’ll be exploring your system’s process tree in the next section.

Note: The precise mechanism for creating processes differs depending on the operating system. For a brief overview, the Wikipedia article on process management has a short section on process creation.

For more details about the Windows mechanism, check out the win32 API documentation page on creating processes

On UNIX-based systems, processes are typically created by using

>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
8 to copy the current process and then replacing the child process with one of the
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
9 family of functions.

The parent-child relationship between a process and its subprocess isn’t always the same. Sometimes the two processes will share specific resources, like inputs and outputs, but sometimes they won’t. Sometimes child processes live longer than the parent. A child outliving the parent can lead to orphaned or zombie processes, though more discussion about those is outside the scope of this tutorial.

When a process has finished running, it’ll usually end. Every process, on exit, should return an integer. This integer is referred to as the return code or exit status. Zero is synonymous with success, while any other value is considered a failure. Different integers can be used to indicate the reason why a process has failed.

In the same way that you can return a value from a function in Python, the operating system expects an integer return value from a process once it exits. This is why the canonical C

>>> shlex.split("echo 'Hello, World!'")
['echo', 'Hello, World!']
0 function usually returns an integer:

// minimal_program.c

int main(){
    return 0;
}

This example shows a minimal amount of C code necessary for the file to compile with

>>> shlex.split("echo 'Hello, World!'")
['echo', 'Hello, World!']
1 without any warnings. It has a
>>> shlex.split("echo 'Hello, World!'")
['echo', 'Hello, World!']
0 function that returns an integer. When this program runs, the operating system will interpret its execution as successful since it returns zero.

So, what processes are running on your system right now? In the next section, you’ll explore some of the tools that you can use to take a peek at your system’s process tree. Being able to see what processes are running and how they’re structured will come in handy when visualizing how the

$ python timer.py 5
0 module works.

Remove ads

Active Processes on Your System

You may be curious to see what processes are running on your system right now. To do that, you can use platform-specific utilities to track them:

  • Windows
  • Linux + macOS

There are many tools available for Windows, but one which is easy to get set up, is fast, and will show you the process tree without much effort is Process Hacker.

You can install Process Hacker by going to the downloads page or with Chocolatey:

PS> choco install processhacker

Open the application, and you should immediately see the process tree.

One of the native commands that you can use with PowerShell is

>>> shlex.split("echo 'Hello, World!'")
['echo', 'Hello, World!']
4, which lists the active processes on the command line.
>>> shlex.split("echo 'Hello, World!'")
['echo', 'Hello, World!']
5 is a command prompt utility that does the same.

The official Microsoft version of Process Hacker is part of the Sysinternals utilities, namely Process Monitor and Process Explorer. You also get PsList, which is a command-line utility similar to

>>> shlex.split("echo 'Hello, World!'")
['echo', 'Hello, World!']
6 on UNIX. You can install Sysinternals by going to the downloads page or by using Chocolatey:

PS> choco install sysinternals

You can also use the more basic, but classic, Task Manager—accessible by pressing Win+X and selecting the Task Manager.

For UNIX-based systems, there are many command-line utilities to choose from:

  • >>> shlex.split("echo 'Hello, World!'")
    ['echo', 'Hello, World!']
    
    7: The classic process and resource monitor, often installed by default. Once it’s running, to see the tree view, also called the forest view, press Shift+V. The forest view may not work on the default macOS
    >>> shlex.split("echo 'Hello, World!'")
    ['echo', 'Hello, World!']
    
    7.
  • >>> shlex.split("echo 'Hello, World!'")
    ['echo', 'Hello, World!']
    
    9: More advanced and user-friendly version of
    >>> shlex.split("echo 'Hello, World!'")
    ['echo', 'Hello, World!']
    
    7.
  • >>> subprocess.run(["notepad"])
    CompletedProcess(args=['notepad'], returncode=0)
    
    1: Another version of
    >>> shlex.split("echo 'Hello, World!'")
    ['echo', 'Hello, World!']
    
    7 with more information, but more technical.
  • >>> subprocess.run(["notepad"])
    CompletedProcess(args=['notepad'], returncode=0)
    
    3: A Python implementation of
    >>> shlex.split("echo 'Hello, World!'")
    ['echo', 'Hello, World!']
    
    7 with nice visuals.
  • >>> shlex.split("echo 'Hello, World!'")
    ['echo', 'Hello, World!']
    
    6: A utility specifically to explore the process tree.

On macOS, you also have the Activity Monitor application in your utilities. In the View menu, if you select All Processes, Hierarchically, you should be able to see your process tree.

You can also explore the Python psutil library, which allows you to retrieve running process information on both Windows and UNIX-based systems.

One universal attribute of process tracking across systems is that each process has a process identification number, or PID, which is a unique integer to identify the process within the context of the operating system. You’ll see this number on most of the utilities listed above.

Along with the PID, it’s typical to see the resource usage, such as CPU percentage and amount of RAM that a particular process is using. This is the information that you look for if a program is hogging all your resources.

The resource utilization of processes can be useful for developing or debugging scripts that use the

$ python timer.py 5
0 module, even though you don’t need the PID, or any information about what resources processes are using in the code itself. While playing with the examples that are coming up, consider leaving a representation of the process tree open to see the new processes pop up.

You now have a bird’s-eye view of processes. You’ll deepen your mental model throughout the tutorial, but now it’s time to see how to start your own processes with the Python

$ python timer.py 5
0 module.

Overview of the Python $ python timer.py 5 0 Module

The Python

$ python timer.py 5
0 module is for launching child processes. These processes can be anything from GUI applications to the shell. The parent-child relationship of processes is where the sub in the
$ python timer.py 5
0 name comes from. When you use
$ python timer.py 5
0, Python is the parent that creates a new child process. What that new child process is, is up to you.

Python

$ python timer.py 5
0 was originally proposed and accepted for Python 2.4 as an alternative to using the
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
3 module. Some documented changes have happened as late as 3.8. The examples in this article were tested with Python 3.10.4, but you only need 3.8+ to follow along with this tutorial.

Most of your interaction with the Python

$ python timer.py 5
0 module will be via the
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 function. This blocking function will start a process and wait until the new process exits before moving on.

The documentation recommends using

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 for all cases that it can handle. For edge cases where you need more control, the
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 class can be used.
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 is the underlying class for the whole
$ python timer.py 5
0 module. All functions in the
$ python timer.py 5
0 module are convenience wrappers around the
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor and its instance methods. Near the end of this tutorial, you’ll dive into the
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 class.

Note: If you’re trying to decide whether you need

$ python timer.py 5
0 or not, check out the section on deciding whether you need
$ python timer.py 5
0 for your task.

You may come across other functions like

PS> choco install processhacker
05,
PS> choco install processhacker
06, and
PS> choco install processhacker
07, but these belong to the older
$ python timer.py 5
0 API from Python 3.5 and earlier. Everything these three functions do can be replicated with the newer
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 function. The older API is mainly still there for backwards compatibility, and you won’t cover it in this tutorial.

There’s also a fair amount of redundancy in the

$ python timer.py 5
0 module, meaning that there are various ways to achieve the same end goal. You won’t be exploring all variations in this tutorial. What you will find, though, are robust techniques that should keep you on the right path.

Basic Usage of the Python $ python timer.py 5 0 Module

In this section, you’ll take a look at some of the most basic examples demonstrating the usage of the

$ python timer.py 5
0 module. You’ll start by exploring a bare-bones command-line timer program with the
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 function.

If you want to follow along with the examples, then create a new folder. All the examples and programs can be saved in this folder. Navigate to this newly created folder on the command line in preparation for the examples coming up. All the code in this tutorial is standard library Python—with no external dependencies required—so a virtual environment isn’t necessary.

Remove ads

The Timer Example

To come to grips with the Python

$ python timer.py 5
0 module, you’ll want a bare-bones program to run and experiment with. For this, you’ll use a program written in Python:

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")

The timer program uses

PS> choco install processhacker
15 to accept an integer as an argument. The integer represents the number of seconds that the timer should wait until exiting, which the program uses
PS> choco install processhacker
16 to achieve. It’ll play a small animation representing each passing second until it exits:

It’s not much, but the key is that it serves as a cross-platform process that runs for a few seconds and which you can easily tinker with. You’ll be calling it with

$ python timer.py 5
0 as if it were a separate executable.

Note: Calling Python programs with the Python

$ python timer.py 5
0 module doesn’t make much sense—there’s usually no need for other Python modules to be in separate processes since you can just import them.

The main reason you’ll be using Python programs for most of the examples in this tutorial is that they’re cross-platform, and you most likely already have Python installed!

You may be tempted to think that starting a new process could be a neat way to achieve concurrency, but that’s not the intended use case for the

$ python timer.py 5
0 module. Maybe what you need are other Python modules dedicated to concurrency, covered in a later section.

The

$ python timer.py 5
0 module is mainly for calling programs other than Python. But, as you can see, you can call Python too if you want! For more discussion on the use cases of
$ python timer.py 5
0, check out the section where this is discussed in more depth, or one of the later examples.

Okay, ready to get stuck in! Once you have the

PS> choco install processhacker
22 program ready, open a Python interactive session and call the timer with
$ python timer.py 5
0:

>>>

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)

With this code, you should’ve seen the animation playing right in the REPL. You imported

$ python timer.py 5
0 and then called the
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 function with a list of strings as the one and only argument. This is the
PS> choco install processhacker
26 parameter of the
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 function.

On executing

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5, the timer process starts, and you can see its output in real time. Once it’s done, it returns an instance of the
PS> choco install processhacker
29 class.

On the command line, you might be used to starting a program with a single string:

$ python timer.py 5

However, with

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 you need to pass the command as a sequence, as shown in the
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 example. Each item in the sequence represents a token which is used for a system call to start a new process.

Note: Calling

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 isn’t the same as calling programs on the command line. The
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 function makes a system call, foregoing the need for a shell. You’ll cover interaction with the shell in a later section.

Shells typically do their own tokenization, which is why you just write the commands as one long string on the command line. With the Python

$ python timer.py 5
0 module, though, you have to break up the command into tokens manually. For instance, executable names, flags, and arguments will each be one token.

Note: You can use the

PS> choco install processhacker
35 module to help you out if you need, just bear in mind that it’s designed for POSIX compliant systems and may not work well in Windows environments:

>>>

>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)

The

PS> choco install processhacker
36 function divides a typical command into the different tokens needed. The
PS> choco install processhacker
35 module can come in handy when it may be not obvious how to divide up more complex commands that have special characters, like spaces:

>>>

>>> shlex.split("echo 'Hello, World!'")
['echo', 'Hello, World!']

You’ll note that the message, which contains spaces, is preserved as a single token, and the extra quotation marks are no longer needed. The extra quotation marks on the shell serve to group the token together, but since

$ python timer.py 5
0 uses sequences, it’s always unambiguous which parts should be interpreted as one token.

Now that you’re familiar with some of the very basics of starting new processes with the Python

$ python timer.py 5
0 module, coming up you’ll see that you can run any kind of process, not just Python or text-based programs.

The Use of $ python timer.py 5 0 to Run Any App

With

$ python timer.py 5
0, you aren’t limited to text-based applications like the shell. You can call any application that you can with the Start menu or app bar, as long as you know the precise name or path of the program that you want to run:

  • Windows
  • Linux
  • macOS

>>>

>>> subprocess.run(["notepad"])
CompletedProcess(args=['notepad'], returncode=0)

>>>

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)

Depending on your Linux distribution, you may have a different text editor, such as

PS> choco install processhacker
42,
PS> choco install processhacker
43,
PS> choco install processhacker
44, or
PS> choco install processhacker
45.

>>>

PS> choco install processhacker
0

These commands should open up a text editor window. Usually

PS> choco install processhacker
29 won’t get returned until you close the editor window. Yet in the case of macOS, since you need to run the launcher process
PS> choco install processhacker
47 to launch TextEdit, the
PS> choco install processhacker
29 gets returned straight away.

Launcher processes are in charge of launching a specific process and then ending. Sometimes programs, such as web browsers, have them built in. The mechanics of launcher processes is out of the scope of this tutorial, but suffice to say that they’re able to manipulate the operating system’s process tree to reassign parent-child relationships.

Note: There are many problems that you might initially reach for

$ python timer.py 5
0 to solve, but then you’ll find a specific module or library that solves it for you. This tends to be a theme with
$ python timer.py 5
0 since it is quite a low-level utility.

An example of something that you might want to do with

$ python timer.py 5
0 is to open a web browser to a specific page. However, for that, it’s probably best to use the Python module
PS> choco install processhacker
52. The
PS> choco install processhacker
52 module uses
$ python timer.py 5
0 under the hood but handles all the finicky cross-platform and browser differences that you might encounter.

Then again,

$ python timer.py 5
0 can be a remarkably useful tool to get something done quickly. If you don’t need a full-fledged library, then
$ python timer.py 5
0 can be your Swiss Army knife. It all depends on your use case. More discussion on this topic will come later.

You’ve successfully started new processes using Python! That’s

$ python timer.py 5
0 at its most basic. Next up, you’ll take a closer look at the
PS> choco install processhacker
29 object that’s returned from
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5.

Remove ads

The PS> choco install processhacker 29 Object

When you use

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5, the return value is an instance of the
PS> choco install processhacker
29 class. As the name suggests,
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 returns the object only once the child process has ended. It has various attributes that can be helpful, such as the
PS> choco install processhacker
26 that were used for the process and the
PS> choco install processhacker
65.

To see this clearly, you can assign the result of

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 to a variable, and then access its attributes such as
PS> choco install processhacker
67:

>>>

PS> choco install processhacker
1

The process has a return code that indicates failure, but it doesn’t raise an exception. Typically, when a

$ python timer.py 5
0 process fails, you’ll always want an exception to be raised, which you can do by passing in a
PS> choco install processhacker
69 argument:

>>>

PS> choco install processhacker
2

There are various ways to deal with failures, some of which will be covered in the next section. The important point to note for now is that

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 won’t necessarily raise an exception if the process fails unless you’ve passed in a
PS> choco install processhacker
69 argument.

The

PS> choco install processhacker
29 also has a few attributes relating to input/output (I/O), which you’ll cover in more detail in the communicating with processes section. Before communicating with processes, though, you’ll learn how to handle errors when coding with
$ python timer.py 5
0.

$ python timer.py 5 0 Exceptions

As you saw earlier, even if a process exits with a return code that represents failure, Python won’t raise an exception. For most use cases of the

$ python timer.py 5
0 module, this isn’t ideal. If a process fails, you’ll usually want to handle it somehow, not just carry on.

A lot of

$ python timer.py 5
0 use cases involve short personal scripts that you might not spend much time on, or at least shouldn’t spend much time on. If you’re tinkering with a script like this, then you’ll want
$ python timer.py 5
0 to fail early and loudly.

PS> choco install processhacker 78 for Non-Zero Exit Code

If a process returns an exit code that isn’t zero, you should interpret that as a failed process. Contrary to what you might expect, the Python

$ python timer.py 5
0 module does not automatically raise an exception on a non-zero exit code. A failing process is typically not something you want your program to pass over silently, so you can pass a
PS> choco install processhacker
69 argument to
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 to raise an exception:

>>>

PS> choco install processhacker
2

The

PS> choco install processhacker
78 is raised as soon as the subprocess runs into a non-zero return code. If you’re developing a short personal script, then perhaps this is good enough for you. If you want to handle errors more gracefully, then read on to the section on exception handling.

One thing to bear in mind is that the

PS> choco install processhacker
78 does not apply to processes that may hang and block your execution indefinitely. To guard against that, you’d want to take advantage of the
PS> choco install processhacker
84 parameter.

PS> choco install processhacker 85 for Processes That Take Too Long

Sometimes processes aren’t well behaved, and they might take too long or just hang indefinitely. To handle those situations, it’s always a good idea to use the

PS> choco install processhacker
84 parameter of the
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 function.

Passing a

PS> choco install processhacker
88 argument to
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 will cause the function to shut down the process and raise a
PS> choco install processhacker
85 error after one second:

>>>

PS> choco install processhacker
4

In this example, the first dot of the timer animation was output, but the subprocess was shut down before being able to complete.

The other type of error that might happen is if the program doesn’t exist on that particular system, which raises one final type of error.

Remove ads

PS> choco install processhacker 91 for Programs That Don’t Exist

The final type of exception you’ll be looking at is the

PS> choco install processhacker
91, which is raised if you try and call a program that doesn’t exist on the target system:

>>>

PS> choco install processhacker
5

This type of error is raised no matter what, so you don’t need to pass in any arguments for the

PS> choco install processhacker
91.

Those are the main exceptions that you’ll run into when using the Python

$ python timer.py 5
0 module. For many use cases, knowing the exceptions and making sure that you use
PS> choco install processhacker
84 and
PS> choco install processhacker
96 arguments will be enough. That’s because if the subprocess fails, then that usually means that your script has failed.

However, if you have a more complex program, then you may want to handle errors more gracefully. For instance, you may need to call many processes over a long period of time. For this, you can use the

PS> choco install processhacker
97 …
PS> choco install processhacker
98 construct.

An Example of Exception Handling

Here’s a code snippet that shows the main exceptions that you’ll want to handle when using

$ python timer.py 5
0:

PS> choco install processhacker
6

This snippet shows you an example of how you might handle the three main exceptions raised by the

$ python timer.py 5
0 module.

Now that you’ve used

$ python timer.py 5
0 in its basic form and handled some exceptions, it’s time to get familiar with what it takes to interact with the shell.

Introduction to the Shell and Text-Based Programs With $ python timer.py 5 0

Some of the most popular use cases of the

$ python timer.py 5
0 module are to interact with text-based programs, typically available on the shell. That’s why in this section, you’ll start to explore all the moving parts involved when interacting with text-based programs, and perhaps question if you need the shell at all!

The shell is typically synonymous with the command-line interface or CLI, but this terminology isn’t entirely accurate. There are actually two separate processes that make up the typical command-line experience:

  1. The interpreter, which is typically thought of as the whole CLI. Common interpreters are Bash on Linux, Zsh on macOS, or PowerShell on Windows. In this tutorial, the interpreter will be referred to as the shell.
  2. The interface, which displays the output of the interpreter in a window and sends user keystrokes to the interpreter. The interface is a separate process from the shell, sometimes called a terminal emulator.

When on the command line, it’s common to think that you’re interacting directly with the shell, but you’re really interacting with the interface. The interface takes care of sending your commands to the shell and displaying the shell’s output back to you.

With this important distinction in mind, it’s time to turn your attention to what

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 is actually doing. It’s common to think that calling
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 is somehow the same as typing a command in a terminal interface, but there are important differences.

While all new process are created with the same system calls, the context from which the system call is made is different. The

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 function can make a system call directly and doesn’t need to go through the shell to do so:

In fact, many programs that are thought of as shell programs, such as Git, are really just text-based programs that don’t need a shell to run. This is especially true of UNIX environments, where all of the familiar utilities like

$ python timer.py 5
3,
PS> choco install sysinternals
08,
PS> choco install sysinternals
09, and
PS> choco install sysinternals
10 are actually separate executables that can be called directly:

>>>

PS> choco install processhacker
7

There are some tools that are specific to shells, though. Finding tools embedded within the shell is far more common on Windows shells like PowerShell, where commands like

$ python timer.py 5
3 are part of the shell itself and not separate executables like they are in a UNIX environment:

>>>

PS> choco install processhacker
8

In PowerShell,

$ python timer.py 5
3 is the default alias for
PS> choco install sysinternals
13, but calling that won’t work either because
PS> choco install sysinternals
13 isn’t a separate executable—it’s part of PowerShell itself.

The fact that many text-based programs can operate independently from the shell may make you wonder if you can cut out the middle process—namely, the shell—and use

$ python timer.py 5
0 directly with the text-based programs typically associated with the shell.

Remove ads

Use Cases for the Shell and $ python timer.py 5 0

There are a few common reasons why you might want to call the shell with the

PS> choco install sysinternals
17 subprocess module:

  • When you know certain commands are only available via the shell, which is more common in Windows
  • When you’re experienced in writing shell scripts with a particular shell, so you want to leverage your ability there to do certain tasks while still working primarily in Python
  • When you’ve inherited a large shell script that might do nothing that Python couldn’t do, but would take a long time to reimplement in Python

This isn’t an exhaustive list!

You might use the shell to wrap programs or to do some text processing. However, the syntax can be very cryptic when compared to Python. With Python, text processing workflows are easier to write, easier to maintain, generally more performant, and cross-platform to boot. So it’s well worth considering going without the shell.

What often happens, though, is that you just don’t have the time or it’s not worth the effort to reimplement existing shell scripts in Python. In those cases, using

$ python timer.py 5
0 for some sloppy Python isn’t a bad thing!

Common reasons for using

$ python timer.py 5
0 itself are similar in nature to using the shell with
$ python timer.py 5
0:

  • When you have to use or analyze a black box, or even a white box
  • When you want a wrapper for an application
  • When you need to launch another application
  • As an alternative to basic shell scripts

Note: A black box could be a program that can be freely used but whose source code isn’t available, so there’s no way to know exactly what it does and no way to modify its internals.

Similarly, a white box could be a program whose source code is available but can’t be changed. It could also be a program whose source code you could change, but its complexity means that it would take you a long time to get your head around it to be able to change it.

In these cases, you can use

$ python timer.py 5
0 to wrap your boxes of varying opacity, bypassing any need to change or reimplement things in Python.

Often you’ll find that for

$ python timer.py 5
0 use cases, there will be a dedicated library for that task. Later in the tutorial, you’ll examine a script that creates a Python project, complete with a virtual environment and a fully initialized Git repository. However, the Cookiecutter and Copier libraries already exist for that purpose.

Even though specific libraries might be able to do your task, it may still be worth doing things with

$ python timer.py 5
0. For one, it might be much faster for you to execute what you already know how to do, rather than learning a new library.

Additionally, if you’re sharing this script with friends or colleagues, it’s convenient if your script is pure Python without any other dependencies, especially if your script needs to go on minimal environments like servers or embedded systems.

However, if you’re using

$ python timer.py 5
0 instead of
PS> choco install sysinternals
25 to read and write a few files with Bash, you might want to consider learning how to read and write with Python. Learning how to read and write files doesn’t take long, and it’ll definitely be worth it for such a common task.

With that out of the way, it’s time to get familiar with the shell environments on both Windows and UNIX-based systems.

Basic Usage of $ python timer.py 5 0 With UNIX-Based Shells

To run a shell command using

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5, the
PS> choco install processhacker
26 should contain the shell that you want to use, the flag to indicate that you want it to run a specific command, and the command that you’re passing in:

>>>

PS> choco install processhacker
9

Here a common shell command is demonstrated. It uses

$ python timer.py 5
3 piped into
PS> choco install sysinternals
09 to filter some of the entries. The shell is handy for this kind of operation because you can take advantage of the pipe operator (
PS> choco install sysinternals
31). You’ll cover pipes in more detail later.

You can replace

PS> choco install sysinternals
32 with the shell of your choice. The
PS> choco install sysinternals
33 flag stands for command, but may be different depending on the shell that you’re using. This is almost the exact equivalent of what happens when you add the
PS> choco install sysinternals
34 argument:

>>>

PS> choco install sysinternals
0

The

PS> choco install sysinternals
34 argument uses
PS> choco install sysinternals
36 behind the scenes, so it’s almost the equivalent of the previous example.

Note: On UNIX-based systems, the

PS> choco install sysinternals
37 shell was traditionally the Bourne shell. That said, the Bourne shell is now quite old, so many operating systems use
PS> choco install sysinternals
37 as a link to Bash or Dash.

This can often be different from the shell used with the terminal interface that you interact with. For instance, since macOS Catalina, the default shell that you’ll find on the command-line app has changed from Bash to Zsh, yet

PS> choco install sysinternals
37 often still points to Bash. Likewise, on Ubuntu,
PS> choco install sysinternals
37 points to Dash, but the default that you typically interact with on the command-line application is still Bash.

So, calling

PS> choco install sysinternals
37 on your system may result in a different shell than what is found in this tutorial. Nevertheless, the examples should all still work.

You’ll note that the token after

PS> choco install sysinternals
42 should be one single token, with all the spaces included. Here you’re giving control to the shell to parse the command. If you were to include more tokens, this would be interpreted as more options to pass to the shell executable, not as additional commands to run inside the shell.

Remove ads

Basic Usage of $ python timer.py 5 0 With Windows Shells

In this section, you’ll cover basic use of the shell with

$ python timer.py 5
0 in a Windows environment.

To run a shell command using

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5, the
PS> choco install processhacker
26 should contain the shell that you want to use, the flag to indicate that you want it to run a specific command, and the command that you’re passing in:

>>>

PS> choco install sysinternals
1

Note that

PS> choco install sysinternals
47 and
PS> choco install sysinternals
48 both work. If you don’t have PowerShell Core, then you can call
PS> choco install sysinternals
49 or
PS> choco install sysinternals
50.

You’ll note that the token after

PS> choco install sysinternals
51 should be one single token, with all the spaces included. Here you’re giving control to the shell to parse the command. If you were to include more tokens, this would be interpreted as more options to pass to the shell executable, not as additional commands to run inside the shell.

If you need the Command Prompt, then the executable is

PS> choco install sysinternals
52 or
PS> choco install sysinternals
53, and the flag to indicate that the following token is a command is
PS> choco install sysinternals
54:

>>>

PS> choco install sysinternals
2

This last example is the exact equivalent of calling

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 with
PS> choco install sysinternals
34. Said in another way, using the
PS> choco install sysinternals
34 argument is like prepending
PS> choco install sysinternals
58 and
PS> choco install sysinternals
59 to your argument list.

Note: Windows’ evolution has been very different from that of UNIX-based systems. The most widely known shell is the Windows Command Prompt which is by now a legacy shell. The Command Prompt was made to emulate the pre-Windows MS-DOS environment. Many shell scripts, or batch

PS> choco install sysinternals
60 scripts, were written for this environment which are still in use today.

The

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 function with the
PS> choco install sysinternals
62 parameter will almost always end up using the Command Prompt. The
$ python timer.py 5
0 module uses the Windows
PS> choco install sysinternals
64 environment variable, which in almost all cases will point to
PS> choco install sysinternals
53, the Command Prompt. By now, there are so many programs that equate
PS> choco install sysinternals
64 to
PS> choco install sysinternals
53 that changing it would cause much breakage in unexpected places! So, changing
PS> choco install sysinternals
64 is generally not advised.

At this point, you should know about an important security concern that you’ll want to be aware of if you have user-facing elements in your Python program, regardless of the operating system. It’s a vulnerability that’s not confined to

$ python timer.py 5
0. Rather, it can be exploited in many different areas.

A Security Warning

If at any point you plan to get user input and somehow translate that to a call to

$ python timer.py 5
0, then you have to be very careful of injection attacks. That is, take into account potential malicious actors. There are many ways to cause havoc if you just let people run code on your machine.

To use a very simplistic example, where you take user input and send it, unfiltered, to subprocess to run on the shell:

  • Windows
  • Linux + macOS

PS> choco install sysinternals
3

PS> choco install sysinternals
4

You can imagine the intended use case is to wrap

$ python timer.py 5
3 and add something to it. So the expected user behavior is to provide a path like
PS> choco install sysinternals
72. However, if a malicious actor realized what was happening, they could execute almost any code they wanted. Take the following, for instance, but be careful with this:

  • Windows
  • Linux + macOS

PS> choco install sysinternals
73

PS> choco install sysinternals
74

Again, beware! These innocent-looking lines could try and delete everything on the system! In this case the malicious part is in quotes, so it won’t run, but if the quotes were not there, you’d be in trouble. The key part that does this is the call to

PS> choco install sysinternals
08 with the relevant flags to recursively delete all files, folders, and subfolders, and it’ll work to force the deletion through. It can run the
PS> choco install sysinternals
76 and potentially the
PS> choco install sysinternals
08 as entirely separate commands by adding semicolons, which act as command separators allowing what would usually be multiple lines of code to run on one line.

Running these malicious commands would cause irreparable damage to the file system, and would require reinstalling the operating system. So, beware!

Luckily, the operating system wouldn’t let you do this to some particularly important files. The

PS> choco install sysinternals
08 command would need to use
PS> choco install sysinternals
79 in UNIX-based systems, or be run as an administrator in Windows to be completely successful in its mayhem. The command would probably delete a lot of important stuff before stopping, though.

So, make sure that if you’re dynamically building user inputs to feed into a

$ python timer.py 5
0 call, then you’re very careful! With that warning, coming up you’ll be covering using the outputs of commands and chaining commands together—in short, how to communicate with processes once they’ve started.

Remove ads

Communication With Processes

You’ve used the

$ python timer.py 5
0 module to execute programs and send basic commands to the shell. But something important is still missing. For many tasks that you might want to use
$ python timer.py 5
0 for, you might want to dynamically send inputs or use the outputs in your Python code later.

To communicate with your process, you first should understand a little bit about how processes communicate in general, and then you’ll take a look at two examples to come to grips with the concepts.

The Standard I/O Streams

A stream at its most basic represents a sequence of elements that aren’t available all at once. When you read characters and lines from a file, you’re working with a stream in the form of a file object, which at its most basic is a file descriptor. File descriptors are often used for streams. So, it’s not uncommon to see the terms stream, file, and file-like used interchangeably.

When processes are initialized, there are three special streams that a process makes use of. A process does the following:

  1. Reads
    PS> choco install sysinternals
    
    83 for input
  2. Writes to
    PS> choco install sysinternals
    
    84 for general output
  3. Writes to
    PS> choco install sysinternals
    
    85 for error reporting

These are the standard streams—a cross-platform pattern for process communication.

Sometimes the child process inherits these streams from the parent. This is what’s happening when you use

PS> choco install sysinternals
86 in the REPL and are able to see the output of the command. The
PS> choco install sysinternals
84 of the Python interpreter is inherited by the subprocess.

When you’re in a REPL environment, you’re looking at a command-line interface process, complete with the three standard I/O streams. The interface has a shell process as a child process, which itself has a Python REPL as a child. In this situation, unless you specify otherwise,

PS> choco install sysinternals
83 comes from the keyboard, while
PS> choco install sysinternals
84 and
PS> choco install sysinternals
85 are displayed on-screen. The interface, the shell, and the REPL share the streams:

파이썬 subprocess - paisseon subprocess

You can think of the standard I/O streams as byte dispensers. The subprocess fills up

PS> choco install sysinternals
84 and
PS> choco install sysinternals
85, and you fill up
PS> choco install sysinternals
83. Then you read the bytes in
PS> choco install sysinternals
84 and
PS> choco install sysinternals
85, and the subprocess reads from
PS> choco install sysinternals
83.

As with a dispenser, you can stock

PS> choco install sysinternals
83 before it gets linked up to a child process. The child process will then read from
PS> choco install sysinternals
83 as and when it needs to. Once a process has read from a stream, though, the bytes are dispensed. You can’t go back and read them again:

These three streams, or files, are the basis for communicating with your process. In the next section, you’ll start to see this in action by getting the output of a magic number generator program.

The Magic Number Generator Example

Often, when using the

$ python timer.py 5
0 module, you’ll want to use the output for something and not just display the output as you have been doing so far. In this section, you’ll use a magic number generator that outputs, well, a magic number.

Imagine that the magic number generator is some obscure program, a black box, inherited across generations of sysadmins at your job. It outputs a magic number that you need for your secret calculations. You’ll read from the

PS> choco install sysinternals
84 of
$ python timer.py 5
0 and use it in your wrapper Python program:

PS> choco install sysinternals
5

Okay, not really so magical. That said, it’s not the magic number generator that you’re interested in—it’s interacting with a hypothetical black box with

$ python timer.py 5
0 that’s interesting. To grab the number generator’s output to use later, you can pass in a
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
03 argument to
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5:

>>>

PS> choco install sysinternals
6

Passing a

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
05 argument of
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
06 to
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 makes the output of the process available at the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
08 attribute of the completed process object. You’ll note that it’s returned as a bytes object, so you need to be mindful of encodings when reading it.

Also note that the

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
08 attribute of the
PS> choco install processhacker
29 is no longer a stream. The stream has been read, and it’s stored as a bytes object in the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
08 attribute.

With the output available, you can use more than one subprocess to grab values and operate on them in your code:

>>>

PS> choco install sysinternals
7

In this example, you start two magic number processes that fetch two magic numbers and then add them together. For now, you rely on the automatic decoding of the bytes object by the

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
12 constructor. In the next section, though, you’ll learn how to decode and encode explicitly.

Remove ads

The Decoding of Standard Streams

Processes communicate in bytes, and you have a few different ways to deal with encoding and decoding these bytes. Beneath the surface,

$ python timer.py 5
0 has a few ways of getting into text mode.

Text mode means that

$ python timer.py 5
0 will try to take care of encoding itself. To do that, it needs to know what character encoding to use. Most of the options for doing this in
$ python timer.py 5
0 will try to use the default encoding. However, you generally want to be explicit about what encoding to use to prevent a bug that would be hard to find in the future.

You can pass a

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
16 argument for Python to take care of encodings using the default encoding. But, as mentioned, it’s always safer to specify the encodings explicitly using the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
17 argument, as not all systems work with the nearly universal UTF-8:

>>>

PS> choco install sysinternals
8

If in text mode, the

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
08 attribute on a
PS> choco install processhacker
29 is now a string and not a bytes object.

You can also decode the bytes returned by calling the

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
20 method on the
PS> choco install sysinternals
84 attribute directly, without requiring text mode at all:

>>>

PS> choco install sysinternals
9

There are other ways to put

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 into text mode. You can also set a
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
06 value for
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
24 or
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
25, which will also put
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 into text mode. This may seem redundant, but much of this is kept for backwards compatibility, seeing as the
$ python timer.py 5
0 module has changed over the years.

Now that you know how to read and decode the output of a process, it’s time to take a look at writing to the input of a process.

Reaction Game Example

In this section, you’ll use

$ python timer.py 5
0 to interact with a command-line game. It’s a basic program that’s designed to test a human’s reaction time. With your knowledge of standard I/O streams, though, you’ll be able to hack it! The source code of the game makes use of the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
29 and
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
30 module:

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
0

The program starts, asks for the user to press enter, and then after a random amount of time will ask the user to press enter again. It measures from the time the message appears to the time the user presses enter, or at least that’s what the game developer thinks:

The

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
31 function will read from
PS> choco install sysinternals
83 until it reaches a newline, which means an Enter keystroke in this context. It returns everything it consumed from
PS> choco install sysinternals
83 except the newline. With that knowledge, you can use
$ python timer.py 5
0 to interact with this game:

>>>

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
1

A reaction time of 0 milliseconds! Not bad! Considering the average human reaction time is around 270 milliseconds, your program is definitely superhuman. Note that the game rounds its output, so 0 milliseconds doesn’t mean it’s instantaneous.

The

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
35 argument passed to
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 is a string consisting of two newlines. The
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
17 parameter is set to
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
38, which puts
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 into text mode. This sets up the process for it to receive the input you that give it.

Before the program starts,

PS> choco install sysinternals
83 is stocked, waiting for the program to consume the newlines it contains. One newline is consumed to start the game, and the next newline is consumed to react to
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
41.

Now that you know what’s happening—namely that

PS> choco install sysinternals
83 can be stocked, as it were—you can hack the program yourself without
$ python timer.py 5
0. If you start the game and then press Enter a few times, that’ll stock up
PS> choco install sysinternals
83 with a few newlines that the program will automatically consume once it gets to the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
31 line. So your reaction time is really only the time it takes for the reaction game to execute
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
46 and consume an input:

The game developer gets wise to this, though, and vows to release another version, which will guard against this exploit. In the meantime, you’ll peek a bit further under the hood of

$ python timer.py 5
0 and learn about how it wires up the standard I/O streams.

Remove ads

Pipes and the Shell

To really understand subprocesses and the redirection of streams, you really need to understand pipes and what they are. This is especially true if you want to wire up two processes together, feeding one

PS> choco install sysinternals
84 into another process’s
PS> choco install sysinternals
83, for instance. In this section, you’ll be coming to grips with pipes and how to use them with the
$ python timer.py 5
0 module.

Introduction to Pipes

A pipe, or pipeline, is a special stream that, instead of having one file handle as most files do, has two. One handle is read-only, and the other is write-only. The name is very descriptive—a pipe serves to pipe a byte stream from one process to another. It’s also buffered, so a process can write to it, and it’ll hold onto those bytes until it’s read, like a dispenser.

You may be used to seeing pipes on the command line, as you did in the section on shells:

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
2

This command tells the shell to create an

$ python timer.py 5
3 process to list all the files in
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
52. The pipe operator (
PS> choco install sysinternals
31) tells the shell to create a pipe from the
PS> choco install sysinternals
84 of the
$ python timer.py 5
3 process and feed it into the
PS> choco install sysinternals
83 of the
PS> choco install sysinternals
09 process. The
PS> choco install sysinternals
09 process filters out all the lines that don’t contain the string
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
59.

Windows doesn’t have

PS> choco install sysinternals
09, but a rough equivalent of the same command would be as follows:

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
3

However, on Windows PowerShell, things work very differently. As you learned in the Windows shell section of this tutorial, the different commands are not separate executables. Therefore, PowerShell is internally redirecting the output of one command into another without starting new processes.

Note: If you don’t have access to a UNIX-based operating system but have Windows 10 or above, then you actually do have access to a UNIX-based operating system! Check out Windows Subsystem for Linux, which will give you access to a fully featured Linux shell.

You can use pipes for different processes on PowerShell, though getting into the intricacies of which ones is outside the scope of this tutorial. For more information on PowerShell pipes, check out the documentation. So, for the rest of the pipe examples, only UNIX-based examples will be used, as the basic mechanism is the same for both systems. They’re not nearly as common on Windows, anyway.

If you want to let the shell take care of piping processes into one another, then you can just pass the whole string as a command into

$ python timer.py 5
0:

>>>

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
4

This way, you can let your chosen shell take care of piping one process into another, instead of trying to reimplement things in Python. This is a perfectly valid choice in certain situations.

Later in the tutorial, you’ll also come to see that you can’t pipe processes directly with

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5. For that, you’ll need the more complicated
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3. Actual piping is demonstrated in Connecting Two Porcesses Together With Pipes, near the end of the tutorial.

Whether you mean to pipe one process into another with the

$ python timer.py 5
0 module or not, the
$ python timer.py 5
0 module makes extensive use of pipes behind the scenes.

The Pipes of $ python timer.py 5 0

The Python

$ python timer.py 5
0 module uses pipes extensively to interact with the processes that it starts. In a previous example, you used the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
05 parameter to be able to access
PS> choco install sysinternals
84:

>>>

PS> choco install sysinternals
6

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
03 is equivalent to explicitly setting the
PS> choco install sysinternals
84 and
PS> choco install sysinternals
85 parameters to the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
73 constant:

>>>

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
6

The

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
74 constant is nothing special. It’s just a number that indicates to
$ python timer.py 5
0 that a pipe should be created. The function then creates a pipe to link up to the
PS> choco install sysinternals
84 of the subprocess, which the function then reads into the
PS> choco install processhacker
29 object’s
PS> choco install sysinternals
84 attribute. By the time it’s a
PS> choco install processhacker
29, it’s no longer a pipe, but a bytes object that can be accessed multiple times.

Note: Pipe buffers have a limited capacity. Depending on the system you are running on, you may easily run into that limit if you plan on holding large quantities of data in the buffer. To work around this limit, you can use normal files.

You can also pass a file object to any of the standard stream parameters:

>>>

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
7

You can’t pass a bytes object or a string directly to the

PS> choco install sysinternals
83 argument, though. It needs to be something file-like.

Note that the

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
81 that gets returned first is from the call to
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
82 which returns the new stream position, which in this case is the start of the stream.

The

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
35 parameter is similar to the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
05 parameter in that it’s a shortcut. Using the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
35 parameter will create a buffer to store the contents of
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
35, and then link the file up to the new process to serve as its
PS> choco install sysinternals
83.

To actually link up two processes with a pipe from within

$ python timer.py 5
0 is something that you can’t do with
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5. Instead, you can delegate the plumbing to the shell, as you did earlier in the Introduction to the Shell and Text Based Programs with
$ python timer.py 5
0 section.

If you needed to link up different processes without delegating any of the work to the shell, then you could do that with the underlying

>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor. You’ll cover
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 in a later section. In the next section, though, you’ll be simulating a pipe with
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 because in most cases, it’s not vital for processes to be linked up directly.

Pipe Simulation With >>> subprocess.run(["gedit"]) CompletedProcess(args=['gedit'], returncode=0) 5

Though you can’t actually link up two processes together with a pipe by using the

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 function, at least not without delegating it to the shell, you can simulate piping by judicious use of the
PS> choco install sysinternals
84 attribute.

If you’re on a UNIX-based system where almost all typical shell commands are separate executables, then you can just set the

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
35 of the second process to the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
08 attribute of the first
PS> choco install processhacker
29:

>>>

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
8

Here the

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
08 attribute of the
PS> choco install processhacker
29 object of
$ python timer.py 5
3 is set to the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
35 of the
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
04. It’s important that it’s set to
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
35 rather than
PS> choco install sysinternals
83. This is because the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
08 attribute isn’t a file-like object. It’s a bytes object, so it can’t be used as an argument to
PS> choco install sysinternals
83.

As an alternative, you can operate directly with files too, setting them to the standard stream parameters. When using files, you set the file object as the argument to

PS> choco install sysinternals
83, instead of using the
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
35 parameter:

>>>

# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
9

As you learned in the previous section, for Windows PowerShell, doing something like this doesn’t make a whole lot of sense because most of the time, these utilities are part of PowerShell itself. Because you aren’t dealing with separate executables, piping becomes less of a necessity. However, the pattern for piping is still the same if something like this needs to be done.

With most of the tools out the way, it’s now time to think about some practical applications for

$ python timer.py 5
0.

Practical Ideas

When you have an issue that you want to solve with Python, sometimes the

$ python timer.py 5
0 module is the easiest way to go, even though it may not be the most correct.

Using

$ python timer.py 5
0 is often tricky to get working across different platforms, and it has inherent dangers. But even though it may involve some sloppy Python, using
$ python timer.py 5
0 can be a very quick and efficient way to solve a problem.

As mentioned, for most tasks you can imagine doing with

$ python timer.py 5
0, there’s usually a library out there that’s dedicated to that specific task. The library will almost certainly use
$ python timer.py 5
0, and the developers will have worked hard to make the code reliable and to cover all the corner cases that can make using
$ python timer.py 5
0 difficult.

So, even though dedicated libraries exist, it can often be simpler to just use

$ python timer.py 5
0, especially if you’re in an environment where you need to limit your dependencies.

In the following sections, you’ll be exploring a couple of practical ideas.

Creating a New Project: An Example

Say you often need to create new local projects, each complete with a virtual environment and initialized as a Git repository. You could reach for the Cookiecutter library, which is dedicated to that task, and that wouldn’t be a bad idea.

However, using Cookiecutter would mean learning Cookiecutter. Imagine you didn’t have much time, and your environment was extremely minimal anyway—all you could really count on was Git and Python. In these cases,

$ python timer.py 5
0 can quickly set up your project for you:

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
0

This is a command-line tool that you can call to start a project. It’ll take care of creating a

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
20 file and a
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
21 file, and then it’ll run a few commands to create a virtual environment, initialize a git repository, and perform your first commit. It’s even cross-platform, opting to use
PS> choco install sysinternals
25 to create the files and folders, which abstracts away the operating system differences.

Could this be done with Cookiecutter? Could you use GitPython for the

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
23 part? Could you use the
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
24 module to create the virtual environment? Yes to all. But if you just need something quick and dirty, using commands you already know, then just using
$ python timer.py 5
0 can be a great option.

Changing Extended Attributes

If you use Dropbox, you may not know that there’s a way to ignore files when syncing. For example, you can keep virtual environments in your project folder and use Dropbox to sync the code, but keep the virtual environment local.

That said, it’s not as easy as adding a

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
26 file. Rather, it involves adding special attributes to files, which can be done from the command line. These attributes are different between UNIX-like systems and Windows:

  • Windows
  • Linux
  • macOS

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
1

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
2

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3

There are some UNIX-based projects, like dropboxignore, that use shell scripts to make it easier to ignore files and folders. The code is relatively complex, and it won’t work on Windows.

With the

$ python timer.py 5
0 module, you can wrap the different shell commands quite easily to come up with your own utility:

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
4

This is a simplified snippet from the author’s dotDropboxIgnore repository. The

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
28 function detects the operating system with the
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
29 module and returns an object that’s an abstraction around the system-specific shell. The code hasn’t implemented the behavior on macOS, so it raises a
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
30 if it detects it’s running on macOS.

The shell object allows you to call an

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
31 method with a list of
PS> choco install sysinternals
25
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
33 objects to set Dropbox to ignore those files.

On the

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
34 class, the constructor tests to see if PowerShell Core is available, and if not, will fall back to the older Windows PowerShell, which is installed by default on Windows 10.

In the next section, you’ll review some of the other modules that might be interesting to keep in mind when deciding whether to use

$ python timer.py 5
0.

Python Modules Associated With $ python timer.py 5 0

When deciding whether a certain task is a good fit for

$ python timer.py 5
0, there are some associated modules that you may want to be aware of.

Before

$ python timer.py 5
0 existed, you could use
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
39 to run commands. However, as with many things that
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
3 was used for before, standard library modules have come to replace
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
3, so it’s mostly used internally. There are hardly any use cases for using
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
3 yourself.

There’s an official documentation page where you can examine some of the old ways to accomplish tasks with

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
3 and learn how you might do the same with
$ python timer.py 5
0.

It might be tempting to think that

$ python timer.py 5
0 can be used for concurrency, and in simple cases, it can be. But, in line with the sloppy Python philosophy, it’s probably only going to be to hack something together quickly. If you want something more robust, then you’ll probably want to start looking at the
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
46 module.

Depending on the task that you’re attempting, you may be able to accomplish it with the

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
47 or
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
48 modules. If everything is written in Python, then these modules are likely your best bet.

The

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
47 module has a high-level API to create and manage subprocesses too, so if you want more control over non-Python parallel processes, that might be one to check out.

Now it’s time to get deep into

$ python timer.py 5
0 and explore the underlying
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 class and its constructor.

The >>> subprocess.run(["gedit"]) CompletedProcess(args=['gedit'], returncode=0) 7 Class

As mentioned, the underlying class for the whole

$ python timer.py 5
0 module is the
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 class and the
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor. Each function in
$ python timer.py 5
0 calls the
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor under the hood. Using the
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor gives you lots of control over the newly started subprocesses.

As a quick summary,

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 is basically the
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 class constructor, some setup, and then a call to the
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
61 method on the newly initialized
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 object. The
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
61 method is a blocking method that returns the
PS> choco install sysinternals
84 and
PS> choco install sysinternals
85 data once the process has ended.

The name of

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 comes from a similar UNIX command that stands for pipe open. The command creates a pipe and then starts a new process that invokes the shell. The
$ python timer.py 5
0 module, though, doesn’t automatically invoke the shell.

The

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 function is a blocking function, which means that interacting dynamically with a process isn’t possible with it. However, the
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor starts a new process and continues, leaving the process running in parallel.

The developer of the reaction game that you were hacking earlier has released a new version of their game, one in which you can’t cheat by loading

PS> choco install sysinternals
83 with newlines:

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
5

Now the program will display a random character, and you need to press that exact character to have the game register your reaction time:

What’s to be done? First, you’ll need to come to grips with using

>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 with basic commands, and then you’ll find another way to exploit the reaction game.

Using >>> import shlex >>> shlex.split("python timer.py 5") ['python', 'timer.py', '5'] >>> subprocess.run(shlex.split("python timer.py 5")) Starting timer of 5 seconds .....Done! CompletedProcess(args=['python', 'timer.py', '5'], returncode=0) 3

Using the

>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor is very similar in appearance to using
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5. If there’s an argument that you can pass to
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5, then you’ll generally be able to pass it to
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3. The fundamental difference is that it’s not a blocking call—rather than waiting until the process is finished, it’ll run the process in parallel. So you need to take this non-blocking nature into account if you want to read the new process’s output:

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
6

This program calls the timer process in a context manager and assigns

PS> choco install sysinternals
84 to a pipe. Then it runs the
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
78 method on the
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 object and reads its
PS> choco install sysinternals
84.

The

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
78 method is a basic method to check if a process is still running. If it is, then
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
78 returns
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
83. Otherwise, it’ll return the process’s exit code.

Then the program uses

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
84 to try and read as many bytes as are available at
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
08.

Note: If you put the

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 object into text mode and then called
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
87 on
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
08, the call to
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
87 would be blocking until it reached a newline. In this case, a newline would coincide with the end of the timer program. This behavior isn’t desired in this situation.

To read as many bytes as are available at that time, disregarding newlines, you need to read with

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
84. It’s important to note that
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
84 is only available on byte streams, so you need to make sure to deal with encodings manually and not use text mode.

The output of this program first prints

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
83 because the process hasn’t yet finished. The program then prints what is available in
PS> choco install sysinternals
84 so far, which is the starting message and the first character of the animation.

After three seconds, the timer hasn’t finished, so you get

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
83 again, along with two more characters of the animation. After another three seconds, the process has ended, so
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
78 produces
# timer.py

from argparse import ArgumentParser
from time import sleep

parser = ArgumentParser()
parser.add_argument("time", type=int)
args = parser.parse_args()
print(f"Starting timer of {args.time} seconds")
for _ in range(args.time):
    print(".", end="", flush=True)
    sleep(1)
print("Done!")
81, and you get the final characters of the animation and
>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
97:

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
7

In this example, you’ve seen how the

>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor works very differently from
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5. In most cases, you don’t need this kind of fine-grained control. That said, in the next sections, you’ll see how you can pipe one process into another, and how you can hack the new reaction game.

Connecting Two Processes Together With Pipes

As mentioned in a previous section, if you need to connect processes together with pipes, you need to use the

>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor. This is mainly because
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 is a blocking call, so by the time the next process starts, the first one has ended, meaning that you can’t directly link up to its
PS> choco install sysinternals
84.

This procedure will only be demonstrated for UNIX systems, because piping in Windows is far less common, as mentioned in the simulating a pipe section:

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
8

In this example, the two processes are started in parallel. They are joined with a common pipe, and the

$ python timer.py 5
03 loop takes care of reading the pipe at
PS> choco install sysinternals
84 to output the lines.

A key point to note is that in contrast to

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5, which returns a
PS> choco install processhacker
29 object, the
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor returns a
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 object. The standard stream attributes of a
PS> choco install processhacker
29 point to bytes objects or strings, but the same attributes of a
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 object point to the actual streams. This allows you to communicate with processes as they’re running.

Whether you really need to pipe processes into one another, though, is another matter. Ask yourself if there’s much to be lost by mediating the process with Python and using

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 exclusively. There are some situations in which you really need
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7, though, such as hacking the new version of the reaction time game.

Interacting Dynamically With a Process

Now that you know you can use

>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 to interact with a process dynamically as it runs, it’s time to turn that knowledge toward exploiting the reaction time game again:

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
9

With this script, you’re taking complete control of the buffering of a process, which is why you pass in arguments such as

$ python timer.py 5
14 to the Python process and
$ python timer.py 5
15 to
$ python timer.py 5
16. These arguments are to ensure that no extra buffering is taking place.

The script works by using a function that’ll search for one of a list of strings by grabbing one character at a time from the process’s

PS> choco install sysinternals
84. As each character comes through, the script will search for the string.

Note: To make this work on both Windows and UNIX-based systems, two strings are searched for: either

$ python timer.py 5
18 or
$ python timer.py 5
19. The Windows-style carriage return along with the typical newline is required on Windows systems.

After the script has found one of the target strings, which in this case is the sequence of characters before the target letter, it’ll then grab the next character and write that letter to the process’s

PS> choco install sysinternals
83 followed by a newline:

At one millisecond, it’s not quite as good as the original hack, but it’s still very much superhuman. Well done!

With all this fun aside, interacting with processes using

>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
7 can be very tricky and is prone to errors. First, see if you can use
>>> subprocess.run(["gedit"])
CompletedProcess(args=['gedit'], returncode=0)
5 exclusively before resorting to the
>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor.

If you really need to interact with processes at this level, the

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
47 module has a high-level API to create and manage subprocesses.

The

>>> import subprocess
>>> subprocess.run(["python", "timer.py", "5"])
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
47 subprocess functionality is intended for more complex uses of
$ python timer.py 5
0 where you may need to orchestrate various processes. This might be the case if you’re performing complex processing of many image, video, or audio files, for example. If you’re using
$ python timer.py 5
0 at this level, then you’re probably building a library.

Conclusion

You’ve completed your journey into the Python

$ python timer.py 5
0 module. You should now be able to decide whether
$ python timer.py 5
0 is a good fit for your problem. You should also be able to decide whether you need to invoke the shell. Aside from that, you should be able to run subprocesses and interact with their inputs and outputs.

You should also be able to start exploring the possibilities of process manipulation with the

>>> import shlex
>>> shlex.split("python timer.py 5")
['python', 'timer.py', '5']

>>> subprocess.run(shlex.split("python timer.py 5"))
Starting timer of 5 seconds
.....Done!
CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
3 constructor.

Along the way, you’ve:

  • Learned about processes in general
  • Gone from basic to advanced usage of
    $ python timer.py 5
    
    0
  • Understood how to raise and handle errors when using
    >>> subprocess.run(["gedit"])
    CompletedProcess(args=['gedit'], returncode=0)
    
    5
  • Gotten familiar with shells and their intricacies on both Windows and UNIX-like systems
  • Explored the use cases for
    $ python timer.py 5
    
    0 through practical examples
  • Understood the standard I/O streams and how to interact with them
  • Come to grips with pipes, both in the shell and with
    $ python timer.py 5
    
    0
  • Looked at the
    >>> import shlex
    >>> shlex.split("python timer.py 5")
    ['python', 'timer.py', '5']
    
    >>> subprocess.run(shlex.split("python timer.py 5"))
    Starting timer of 5 seconds
    .....Done!
    CompletedProcess(args=['python', 'timer.py', '5'], returncode=0)
    
    3 constructor and used it for some advanced process communication

You’re now ready to bring a variety of executables into your Pythonic sphere of influence!

Source Code: Click here to download the free source code that you’ll use to get acquainted with the Python

$ python timer.py 5
0 module.

Mark as Completed

🐍 Python Tricks 💌

Get a short & sweet Python Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.

파이썬 subprocess - paisseon subprocess

Send Me Python Tricks »

About Ian Currie

파이썬 subprocess - paisseon subprocess
파이썬 subprocess - paisseon subprocess

Ian is a Python nerd who uses it for everything from tinkering to helping people and companies manage their day-to-day and develop their businesses.

» More about Ian


Each tutorial at Real Python is created by a team of developers so that it meets our high quality standards. The team members who worked on this tutorial are:

파이썬 subprocess - paisseon subprocess

Aldren

파이썬 subprocess - paisseon subprocess

Bartosz

파이썬 subprocess - paisseon subprocess

Geir Arne

파이썬 subprocess - paisseon subprocess

Jim

파이썬 subprocess - paisseon subprocess

Kate

Master Real-World Python Skills With Unlimited Access to Real Python

파이썬 subprocess - paisseon subprocess

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

Master Real-World Python Skills
With Unlimited Access to Real Python

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

What Do You Think?

Rate this article:

Tweet Share Share Email

What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.

Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students. Get tips for asking good questions and get answers to common questions in our support portal.