Website Clone Crawler (with Python code)

Jan 7, 2025#python脚本73

AI Translation

This post is translated from Chinese into English through AI.View Original

AI-generated summary

The script is a website mirroring tool that downloads a webpage and its associated files, organizing them by type in a directory structure. It creates a main folder called "sites" and subfolders based on the domain name. The tool modifies the HTML file to point to the locally saved files and generates a ZIP archive containing all downloaded content. It handles various file types such as images, scripts, and stylesheets, ensuring that links in the HTML are updated to reflect the local file paths.

Main functionality:
Creates a 'sites' folder in the current directory
Creates a subfolder in the 'sites' folder according to the domain name
Downloaded files will be categorized by type (e.g., images folder for images, css folder for stylesheets, etc.)
Automatically modifies links in the HTML file to point to local files
Finally generates a ZIP archive containing all downloaded files

For example, if you download https://example.com:
The HTML file will be saved as sites/examplecom/examplecom.htm
All files will be compressed into sites/examplecom.zip
Images will be stored in the sites/examplecom/images/ directory
CSS files will be stored in the sites/examplecom/css/ directory
JavaScript files will be stored in the sites/examplecom/js/ directory

During execution, it will display in the console:
: Detected file links
Downloaded file save paths
Final locations of the generated HTML and ZIP files