Best practices for robust Excel data processing with Pandas and OpenPyXL
Guide for efficient, safe, and standards-compliant Excel data processing.
Always determine the appropriate engine:
.xlsx: Use openpyxl (Default modern format)..xls: Use xlrd (Legacy format)..csv: Use pandas.read_csv.def read_excel_safe(filepath):
try:
if filepath.lower().endswith('.xlsx'):
return pd.read_csv(filepath, engine='openpyxl')
elif filepath.lower().endswith('.xls'):
return pd.read_csv(filepath, engine='xlrd')
return None
except Exception as e:
print(f"Error: {e}")
return None
```
### 3. Handling Temp Files
Always skip Excel temp files (`~$filename.xlsx`):
```python
if filename.startswith('~$'):
continue
Use index=False unless index has meaning:
df.to_excel("output.xlsx", index=False, engine='openpyxl')
For large datasets (>100k rows), openpyxl can be slow. Consider:
Excel file open by user will be locked.
Solution: Catch PermissionError.
try:
df.to_excel("output.xlsx")
except PermissionError:
print("Error: File is open. Please close Excel and try again.")
File downloaded from internet or corrupted format.
Solution: Catch BadZipFile or ValueError.
CSV might have encoding issues (e.g. non-ASCII characters). Solution: Try list of common encodings.
encodings = ['utf-8', 'utf-8-sig', 'cp1252', 'latin1']
for enc in encodings:
try:
return pd.read_csv(file, encoding=enc)
except:
continue
if df.empty:
print("File has no data")
return
Ensure input file has required columns:
required = ['Name', 'Email']
if not all(col in df.columns for col in required):
print("Missing required columns")
pd.read_excel(..., usecols=['A', 'B']) to reduce RAM usage.dtype={'Phone': str} to avoid losing leading zeros.openpyxl vs xlrd)~$PermissionError (File locked)UnicodeDecodeError (Encoding)df.empty before processingWhen activating this skill, print: "🎯 [SKILL ACTIVATED] excel-processing v1.0.0" "📋 Parameters:" " - Input: [file_path]" " - Operation: [read|write|validate]" " - Expected Rows: [count_if_known]"
Before writing/modifying files: "I'll use excel-processing to [action] on [file]. Proceed? [Y/n]"