Setting up and running Chrome and Selenium on the ubuntu or debian. The guide is based on ubuntu 22.04
Selenium is great for automating web tasks, but keeping a bot running 24/7 on a server can be tricky. By using systemd, you can run your Selenium bot as a background service (daemon), ensuring it runs reliably and restarts on failure. This guide will walk you through the steps to set it up, with a focus on configuring it for a Linux VPS.
Installing Google Chrome
Setting up the virtual environment
Installing necessary packages
Creating the Python script
Setting up the systemd service
Fixing block buffering issues
Accessing logs using journalctl
References
First, update all packages.
sudo apt update
Download the stable Google Chrome package.
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
Install Google Chrome.
sudo apt install -y ./google-chrome-stable_current_amd64.deb
Check the installed Google Chrome version.
google-chrome --version
These steps are not mandatory if you're solely running the Selenium bot on your machine. However, it's recommended if you're working on other projects or need isolated environments.
Lets create our virtual environment.
python3 -m venv venv
Activate the virtual environment.
source venv/bin/activate
Now, install selenium and webdriver-manager.
pip install selenium pip install webdriver-manager
The purpose of webdriver-manger is to simplify the management of binary drivers for different browsers. You can learn more about it in its documentation.
## main.py
from selenium import webdriver
## ---- Use for type hint ---- ##
from selenium.webdriver.chrome.webdriver import WebDriver
## --------------------------- ##
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
def create_chrome_web_driver_connection(headless: bool,
detach:bool,
use_sandbox: bool,
use_dev_shm: bool,
window_width: int = 1052,
window_height: int = 825
) -> WebDriver:
service = Service(ChromeDriverManager().install())
options = Options()
options.add_experimental_option("detach", detach)
options.add_argument(f"--window-size={window_width},{window_height}")
options.add_argument("--disable-extensions")
options.add_argument("--disable-renderer-backgrounding")
options.page_load_strategy = 'normal'
if not use_sandbox:
options.add_argument('--no-sandbox')
if not use_dev_shm:
options.add_argument('--disable-dev-shm-usage')
if headless:
options.add_argument("--headless=new")
driver = webdriver.Chrome(service= service, options=options)
return driver
if "__main__" == __name__:
driver = create_chrome_web_driver_connection(headless= True,
detach= False,
use_sandbox= False,
use_dev_shm= False)
driver.get('https://python.org')
print(driver.title)
driver.close()
options.add_experimental_option("detach", detach) :
options.add_argument(f"--window-size={window_width},{window_height}") :
You can remove this line if you want to.
If you plan to run Selenium in headless mode, make sure to set the window size this way; otherwise, in my experience, the default window size might be too small.You can check your window size with this command driver.get_window_size()
options.add_argument("--disable-extensions") :
options.add_argument("--disable-renderer-backgrounding") :
options.page_load_strategy = 'normal' :
options.add_argument('--no-sandbox') :
options.add_argument('--disable-dev-shm-usage') :
options.add_argument("--headless=new") :
In case your selenium program needs to use environment variables there are two ways in which you can achieve this:
Using a .env file with a library like python-dotenv (the more common/popular method).
Using systemd’s built-in option to set up an Environment File.
For this example, we will use the second option.
First, let's create create a conf.d directory inside the /etc directory.
sudo apt update
Next, create a plain text file(this will be our environment file).
sudo apt update
Now you can add your environment variables to the file we just created.
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
Modify the Python script to use the environment variables.
sudo apt install -y ./google-chrome-stable_current_amd64.deb
You need to create a service file inside this /etc/systemd/system/ directory; this is where systemd units installed by the system administrator should be placed.
google-chrome --version
For this example im going to assume that you are in a VPS and you want to run the service as the root user.
[!WARNING]
The service will run with root (administrator) privileges. This gives it full access to the system, but running as root is usually avoided unless necessary for security reasons.
Add this to the service file.
python3 -m venv venv
This section defines metadata and dependencies for the service.
Description=Selenium Bot Service : Provides a short description of what the service does. In this case, it describes it as the "Selenium Bot Service." This description is used in system logs and by systemctl to identify the service.
After=network.target: This ensures that the service starts only after the network is available. The network.target is a systemd target that indicates basic networking functionality is up.
This section specifies the configuration of the service itself, including how it runs, which user runs it, and what to do if it fails.
User : Specifies the user under which the service will run. Here, it’s set to root.
EnvironmentFile : Specifies a file that contains environment variables used by the service.
WorkingDirectory: Specifies the directory from which the service will run. This is the working directory for the service process. Here, the bot files are stored in /root/selenium_bot/
ExecStart : Defines the command to start the service. Here, it runs the main.py file in /root/selenium_bot/
Restart=on-failure : Configures the service to restart automatically if it exits with a failure (i.e., non-zero exit status). This is useful to ensure that the bot service remains running, even if there are occasional failures.
RestartSec=5s : Specifies the delay between restarts in case of failure. In this case, the service will wait 5 seconds before attempting to restart after a failure.
StandardOutput=journal : Redirects the standard output (stdout) of the service to the systemd journal, which can be viewed using journalctl. This is useful for logging and debugging purposes.
StandardError=journal : Redirects the standard error (stderr) output to the systemd journal. Any errors encountered by the service will be logged and can also be viewed using journalctl
This section defines how and when the service should be enabled or started.
WantedBy=multi-user.target : Specifies the target under which the service should be enabled. In this case, multi-user.target is a systemd target that is reached when the system is in a non-graphical multi-user mode (common in servers). This means the service will be started when the system reaches this target, typically when the system has booted to a multi-user environment.
To learn more about all the possible settings for a systemd service check the references
Let's check that our service file is valid; if everything is okay, nothing should be displayed.
sudo apt update
Reload the systemd configuration, looking for new or modified units(services).
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
Start/Restart your service.
sudo apt install -y ./google-chrome-stable_current_amd64.deb
Stop your service.
google-chrome --version
Check your service status.
python3 -m venv venv
Do this if you want your service to start automatically at boot.
source venv/bin/activate
In python when you run your script in an interactive environment( e.g., when you manually run python3 filename.py in a terminal) Python uses line buffering. This means that output, like the one from a print() statement, is shown immediately.
However, when the Python program is run in a non-interactive environment (this is our case), the output will use block buffering. This means that the program will hold onto its output in a buffer until the buffer is full or the program ends, delaying when you can see logs/output.
You can learn more about how python's output buffering works here.
Since we want to view logs and output in real-time, we can address this issue in two ways.
The python3 docs tells us this.
-u Force the stdout and stderr streams to be unbuffered. This option has no effect on the stdin stream
By using the -u flag, Python operates in a fully unbuffered mode for both stdout and stderr. This means that each byte is sent directly to the terminal (or any output stream like a log file) as soon as it is produced. No buffering takes place at all.
Every character that would typically go to stdout (like from print() statements or errors) is immediately written, byte-by-byte, without waiting for a full line or buffer to accumulate.
To use this option run your script like this:
sudo apt update
If you go with this option make sure to modify the ExecStart inside the service file.
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
In Python, the print() function buffers its output by default, meaning it stores the output in memory and only writes it out when the buffer is full or the program ends. By using flush=True, you can force Python to flush the output immediately after the print() call, ensuring that the output appears right away.
sudo apt install -y ./google-chrome-stable_current_amd64.deb
To view the full log history of your systemd unit (service), use the following command.
google-chrome --version
To monitor logs in real time, use the -f flag. This will show only the most recent journal entries, and continuously print new entries as they are appended to the journal.
python3 -m venv venv
The above is the detailed content of How to Set Up Selenium as a Linux Daemon with systemd. For more information, please follow other related articles on the PHP Chinese website!
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
