Every system that writes a file and expects another system to read it faces a hidden danger — what if the reader picks up the file before the writer finishes? The .PROCESSING pattern solves this with one simple trick: never write directly to the final filename.
The .PROCESSING File Pattern: How to Prevent Systems from Reading Incomplete Files
📅 August 2026 | ⏱ 15 min read | Software Engineering
Imagine you are building a system.
One application writes a file — maybe a report, a data export, a transaction log or a configuration file. Another application watches the output folder and picks up files for processing.
Sounds straightforward, right?
But there is a hidden problem that catches many developers off guard:
What happens if the consumer picks up the file while it is still being written?
The result? Partial data. Corrupted records. Silent failures that are incredibly hard to debug in production.
This is one of the most common yet overlooked problems in file-based integration, batch processing pipelines and ETL systems.
And the solution is surprisingly simple — the Atomic File Write Pattern, often implemented using a temporary extension like .PROCESSING.
In this blog, I will explain the problem in depth, walk through the pattern, show generic code examples and discuss when and why you should use it.
A complete guide to understanding the .PROCESSING file pattern — what problem it solves, how it works, code examples in Java and Python, edge cases, pitfalls and best practices for both producers and consumers.
- What is the race condition problem in file-based systems?
- What real-world failures does this cause?
- What is the Atomic File Write Pattern?
- Why rename and not copy?
- How does it work in one-shot vs streaming writers?
- Generic code examples — Java, Python and Shell
- Edge cases and pitfalls
- Consumer-side best practices
- Common variations of this pattern
- When should you use this pattern?
- Frequently asked questions
📌 Note: This article explains the .PROCESSING file pattern in simple English with practical examples. Whether you are a backend developer, a batch processing engineer or someone building file-based integrations for the first time — this guide covers the concept from scratch.
🧠 What Is the Problem? The Race Condition in File-Based Systems
Let's start with the exact problem this pattern solves.
In many real-world systems, two separate applications communicate through files:
Producer — writes output files (reports, data exports, config files, transaction logs)
Consumer — watches the output folder and picks up files matching a specific pattern (e.g., *.dat, *.json, *.csv)
Now imagine this timeline:
Time 0ms → Producer starts creating "orders_2026.json"
Time 50ms → 30% of content written to disk
Time 80ms → Consumer detects "orders_2026.json" in the folder ← PROBLEM
Time 80ms → Consumer reads the file — gets only 30% of data
Time 150ms → Producer finishes writing — but consumer already moved on
The consumer saw a valid filename, assumed the file was ready and processed incomplete data.
The producer did nothing wrong either — it was still writing.
This is a classic race condition.
👉 The fundamental problem is simple: the file's name looks "ready" before its content actually is. The consumer has no way to distinguish a half-written file from a complete one just by looking at the filename.
💥 What Real-World Failures Does This Cause?
This is not a theoretical problem. It causes real failures in production systems every day.
Batch Processing Systems
A nightly job exports 50,000 records to a file. A downstream scheduler picks it up at 40,000 records. Next morning, 10,000 records are "missing" and nobody knows why.
File Transfer Systems
An SFTP watcher detects a new file and transfers it to another server mid-write. The destination gets a truncated file.
Retail and POS Systems
A store's transaction file is generated for headquarters. The central system picks it up before it is complete, causing reconciliation mismatches.
Configuration Deployment
A config file is pushed to application nodes. A node reads it mid-write and starts with invalid or partial configuration, potentially crashing the application.
Data Pipelines and ETL
An ETL pipeline watches for CSV files to load into a database. A partially written CSV causes schema violations, null errors or duplicate processing on retry.
The worst part:
These bugs are incredibly hard to reproduce. They depend on timing — file size, disk speed, how fast the consumer polls. In development and testing, files are small and writes are instant. The bug only appears in production with larger data volumes.
— Developer Pain Point
✅ The Solution: Atomic File Write Using a Temporary Extension
The idea behind the solution is beautifully simple:
Never write directly to the final filename. Write to a temporary name first, then rename when done.
How It Works
Step 1: Create file as → "orders_2026.json.PROCESSING"
Step 2: Write all content → (consumer ignores this extension)
Step 3: Flush and close → (ensures all bytes are on disk)
Step 4: Rename to → "orders_2026.json"
(consumer now sees and picks it up — 100% complete)
The rename operation on most operating systems (Linux, Windows) is atomic — it either completes fully or not at all. There is no intermediate state where the file is half-renamed.
This guarantees that the moment the consumer sees orders_2026.json, it contains 100% of the data.
👉 The pattern works because the consumer filters by extension. It looks for .json or .csv or .dat files — it never touches .PROCESSING files. So no matter how long the write takes, the consumer will never see an incomplete file.
Simple rule:
If the consumer can see the filename, the file must be complete. If the file is not complete, the consumer must not be able to see it.
— Core Principle
🔄 Why Rename and Not Copy?
You might ask — why not write to a temp file and then copy it to the final name?
The answer is important:
Rename Is Atomic
Rename is a metadata operation at the filesystem level. The OS simply updates the directory entry to point to the new name. It happens instantly, regardless of file size. A 1 KB file and a 10 GB file rename in the same amount of time.
Copy Is NOT Atomic
Copy reads bytes from source and writes to destination. During the copy, the destination file exists but is incomplete — the exact same race condition you were trying to avoid.
👉 Always prefer rename over copy when the source and destination are on the same filesystem or partition. Rename is instant and atomic. Copy recreates the exact problem you are trying to solve.
⏱️ Long-Running Writers vs One-Shot Writers
The .PROCESSING pattern behaves differently depending on how your writer works.
Long-Running Chunk-Oriented Writers
When a system processes data in chunks and writes each chunk to a file incrementally:
Step Start → open() → Create "output.dat.PROCESSING"
Chunk 1 → write() → Append 100 records to .PROCESSING file
Chunk 2 → write() → Append 100 more records
Chunk 3 → write() → Append 100 more records
Step End → close() → Rename to "output.dat"
In this case, the .PROCESSING file can exist for minutes or even hours depending on data volume. You can clearly see it in the filesystem while the job runs.
One-Shot Writers (Collect-Then-Write)
Some writers collect all data in memory and write once at the end:
Chunk 1 → write() → Collect in memory
Chunk 2 → write() → Collect in memory
Chunk 3 → write() → Collect in memory
Step End → flush() → Write "output.json.PROCESSING" → Rename to "output.json"
Here the .PROCESSING file exists only for milliseconds. You may never see it in a file browser.
But the protection is still active.
Even milliseconds matter — a fast polling consumer or a concurrent process could still read an incomplete file without this safeguard.
👉 Whether the .PROCESSING file exists for two hours or two milliseconds, the principle is the same: the consumer never sees an incomplete file. The duration doesn't matter — the atomicity does.
Key insight:
If you add the .PROCESSING pattern and you never see the .PROCESSING file — that means your writes are fast, not that the pattern is broken. Check your application logs to confirm it is working.
— Common Confusion Cleared
☕ Code Example: Simple One-Shot Writer (Java)
Here is a basic utility method that any application can use for safe file writing:
public class SafeFileWriter {
private static final String TEMP_SUFFIX = ".PROCESSING";
/**
* Writes content to a file using the atomic rename pattern.
*
* @param finalPath The intended output file path
* @param content The content to write
*/
public static void writeAtomically(String finalPath, String content)
throws IOException {
File finalFile = new File(finalPath);
File tempFile = new File(finalPath + TEMP_SUFFIX);
// Ensure parent directory exists
File parentDir = finalFile.getParentFile();
if (parentDir != null && !parentDir.exists()) {
parentDir.mkdirs();
}
// Step 1: Write to temporary file
try (BufferedWriter writer = new BufferedWriter(
new OutputStreamWriter(
new FileOutputStream(tempFile), "UTF-8"))) {
writer.write(content);
writer.flush();
}
// Step 2: Atomic rename
if (!tempFile.renameTo(finalFile)) {
tempFile.delete();
throw new IOException(
"Failed to rename " + tempFile + " to " + finalFile);
}
}
}
Usage:
SafeFileWriter.writeAtomically(
"/output/reports/daily_sales_20260824.csv",
csvContent
);
☕ Code Example: Streaming Large Files (Java)
For large files where you write data in chunks or iterations:
public class StreamingSafeFileWriter implements Closeable {
private final File finalFile;
private final File tempFile;
private final BufferedWriter writer;
public StreamingSafeFileWriter(String finalPath) throws IOException {
this.finalFile = new File(finalPath);
this.tempFile = new File(finalPath + ".PROCESSING");
File parentDir = finalFile.getParentFile();
if (parentDir != null && !parentDir.exists()) {
parentDir.mkdirs();
}
this.writer = new BufferedWriter(
new OutputStreamWriter(
new FileOutputStream(tempFile), "UTF-8"));
}
/** Write a single line of data */
public void writeLine(String line) throws IOException {
writer.write(line);
writer.newLine();
}
/** Flush, close, and atomically rename */
@Override
public void close() throws IOException {
writer.flush();
writer.close();
if (!tempFile.renameTo(finalFile)) {
tempFile.delete();
throw new IOException(
"Atomic rename failed: " + tempFile + " → " + finalFile);
}
}
}
Usage:
try (StreamingSafeFileWriter writer =
new StreamingSafeFileWriter("/output/transactions.dat")) {
for (Transaction txn : transactions) {
writer.writeLine(txn.toDelimitedString());
}
}
// File is now safely available as "transactions.dat"
👉 This is exactly the pattern used by enterprise batch frameworks internally. The .PROCESSING file stays open for the entire duration of the step. Only when close() is called does the final file appear — complete and safe to read.
🐍 Code Example: Python
import os
def write_atomically(final_path, content):
"""
Writes content to a file using the atomic rename pattern.
"""
temp_path = final_path + ".PROCESSING"
# Ensure directory exists
os.makedirs(os.path.dirname(final_path), exist_ok=True)
# Write to temporary file
with open(temp_path, 'w', encoding='utf-8') as f:
f.write(content)
f.flush()
os.fsync(f.fileno()) # Force OS to write to disk
# Atomic rename
os.rename(temp_path, final_path)
🐚 Code Example: Shell Script
#!/bin/bash
OUTPUT_FILE="/data/exports/inventory.csv"
TEMP_FILE="${OUTPUT_FILE}.PROCESSING"
# Generate data into temp file
generate_report > "$TEMP_FILE"
# Atomic rename
mv "$TEMP_FILE" "$OUTPUT_FILE"
echo "File ready: $OUTPUT_FILE"
⚠️ Edge Cases and Pitfalls
The pattern is simple, but there are important edge cases that developers miss.
1. Cross-Filesystem Rename Fails
rename() and renameTo() only work atomically when source and destination are on the same filesystem.
If your temp directory is on a different partition or mount point, the OS performs a copy-then-delete instead — which defeats the purpose entirely.
Solution: Always write the .PROCESSING file in the same directory as the final output.
2. Java's renameTo() Can Silently Fail
In Java, File.renameTo() returns false on failure — it does not throw an exception. Many developers forget to check the return value.
// BAD — ignores failure silently
tempFile.renameTo(finalFile);
// GOOD — checks result
if (!tempFile.renameTo(finalFile)) {
throw new IOException("Rename failed!");
}
// BETTER (Java 7+) — throws exception on failure
java.nio.file.Files.move(
tempFile.toPath(),
finalFile.toPath(),
java.nio.file.StandardCopyOption.ATOMIC_MOVE
);
3. Orphan .PROCESSING Files on Failure
If the write fails (disk full, exception, process crash), the .PROCESSING file remains on disk as an orphan. Your system should handle this:
On startup: Scan for leftover .PROCESSING files and either delete them or move them to an error directory.
In the writer: Use a try-finally block to delete the temp file if writing fails.
boolean success = false;
try {
writeContent(tempFile);
success = true;
} finally {
if (!success && tempFile.exists()) {
tempFile.delete(); // Clean up incomplete file
}
}
if (success) {
tempFile.renameTo(finalFile);
}
4. fsync Before Rename
On some operating systems, the content you wrote might still be in the OS buffer cache when you call rename(). If the system crashes after rename but before the cache flushes, you could end up with a renamed but empty or incomplete file.
For truly critical data, call fsync() before renaming:
try (FileOutputStream fos = new FileOutputStream(tempFile)) {
fos.write(content.getBytes("UTF-8"));
fos.flush();
fos.getFD().sync(); // Force write to physical disk
}
tempFile.renameTo(finalFile);
👉 fsync is rarely needed for most applications. But for financial data, medical records or any system where data loss is unacceptable — it adds the final layer of safety between your application and the physical disk.
Remember:
renameTo() in Java returns false on failure — it does NOT throw an exception. Always check the return value. This is one of the most common mistakes developers make with this pattern.
— Common Pitfall
📥 Consumer-Side Best Practices
The .PROCESSING pattern only works if the consumer cooperates. Here are rules for the file-polling side:
1. Filter by Extension
Only pick up files matching the final extension (.csv, .json, .dat). Ignore .PROCESSING, .tmp, .partial and similar extensions.
2. Check File Stability
Before processing, check that the file size has not changed in the last few seconds. This catches cases where the producer forgot to use the pattern.
3. Use Last-Modified Check
Do not process a file that was modified less than N seconds ago. This adds an extra safety margin.
4. Move Before Processing
Once you pick up a file, move it to a work-in-progress directory before processing. This prevents other consumer instances from picking up the same file.
/output/ready/ ← Consumer watches here
/output/in-progress/ ← Consumer moves file here before reading
/output/done/ ← Consumer moves file here after processing
/output/error/ ← Consumer moves file here on failure
👉 A robust file-based integration requires both sides to cooperate. The producer uses .PROCESSING to guarantee completeness. The consumer uses extension filters, stability checks and move-before-read to guarantee safe processing.
🔀 Common Variations of This Pattern
The .PROCESSING extension is just one convention. Different teams and tools use different names, but the concept is identical:
Temp extension — .PROCESSING, .tmp, .partial — Used by enterprise batch systems
Dot-prefix — .output.csv → output.csv — Used by Unix tools and rsync
Separate directory — Write to /staging/, move to /ready/ — Used by ETL pipelines and SFTP systems
PID suffix — file.dat.12345 → file.dat — Used in multi-process environments
Lock file — Create file.dat.lock, write, delete lock — Used by simpler polling systems
Each variation has the same core idea: Signal to consumers that the file is not ready yet.
The naming convention doesn't matter.
Whether you call it .PROCESSING, .tmp, .partial or .inprogress — the pattern is the same. Write to a name the consumer ignores. Rename to a name the consumer recognises. That's the entire trick.
— Pattern Essence
🎯 When Should You Use This Pattern?
Definitely Use When
File watchers or pollers monitor your output directory. Multiple processes or threads write to the same directory. Downstream systems auto-ingest files based on extension or naming patterns. File sizes are large and writes take noticeable time. Data integrity is critical — financial, medical, compliance data. Network or shared filesystems (NFS, SMB) where operations are slower and more prone to partial reads.
Can Skip When
The file is only read by the same process that wrote it, in sequence. You have an explicit signalling mechanism — a separate "ready" flag file, database status column or message queue notification. You are writing to a database or object store that provides its own atomicity guarantees.
👉 When in doubt, use the pattern. It costs almost nothing — two extra lines of code and zero performance overhead (rename is a metadata operation). The cost of not using it — corrupted data in production — is far higher.
📊 Visual Summary: With vs Without the Pattern
Without .PROCESSING
Producer → starts writing "report.csv"
→ 10% written
→ 40% written ← CONSUMER PICKS UP (incomplete data!)
→ 70% written
→ 100% done — but consumer already processed broken data
With .PROCESSING
Producer → starts writing "report.csv.PROCESSING"
→ 10% written
→ 40% written ← Consumer IGNORES (extension doesn't match)
→ 70% written
→ 100% done
→ Rename to "report.csv" ← Consumer picks up — 100% complete ✅
Write to a temporary filename → Complete all writes → Flush to disk → Atomically rename to the final filename.
The consumer never sees an incomplete file.
⏱️ The .PROCESSING Pattern in 30 Seconds
Problem — Consumer reads a file that is still being written.
↓
Root Cause — File has the final name before the content is complete.
↓
Solution — Write to a temporary name (.PROCESSING) that the consumer ignores.
↓
Completion — Rename to the final name only after all content is written and flushed.
↓
Result — Consumer only ever sees 100% complete files.
The easiest sentence to remember:
If the consumer can see the filename, the file must be complete. If the file is not complete, the consumer must not see the filename.
— PrafullTalks
❓ Frequently Asked Questions
1. What is the .PROCESSING file pattern?
It is a technique where a file is first written with a temporary extension like .PROCESSING, and then renamed to its final name only after the write is complete. This prevents downstream systems from reading an incomplete file.
2. Why not just write directly to the final filename?
Because a consumer monitoring the folder might detect and read the file while it is still being written, leading to partial or corrupted data.
3. Is rename really atomic?
Yes, on most operating systems (Linux, Windows, macOS), a rename within the same filesystem is an atomic metadata operation. It either completes fully or not at all.
4. Why is copy not a good alternative to rename?
Copy creates the destination file and writes bytes to it gradually — during the copy, the destination file exists but is incomplete. This recreates the exact same race condition.
5. What happens if the write fails midway?
The .PROCESSING file remains on disk as an orphan. Your application should clean it up — either in a finally block or on startup by scanning for leftover .PROCESSING files.
6. Does it work across different filesystems?
No. Rename is only atomic within the same filesystem or partition. If the temp file and final file are on different partitions, the OS performs a copy-then-delete instead, which is not atomic.
7. I used the pattern but I never see .PROCESSING files — is it working?
Yes. If your writes are fast (one-shot writers that collect data in memory and write once), the .PROCESSING file exists for only milliseconds. Check your application logs to confirm the pattern is executing correctly.
8. What other names are used instead of .PROCESSING?
Common alternatives include .tmp, .partial, .inprogress and dot-prefixed filenames. The naming convention varies by team and framework, but the core pattern is identical.
9. Should I always use this pattern?
Use it whenever another system, process or thread might read your output files. Skip it only when you have an explicit signalling mechanism (like a database flag or message queue) or when the same process reads the file sequentially after writing.
10. Does this pattern have any performance overhead?
Practically none. Rename is a metadata operation — it takes microseconds regardless of file size. The only "overhead" is two extra lines of code.
- The core problem is a race condition — a consumer reads a file before the producer finishes writing it.
- The solution is to write to a temporary name (.PROCESSING) and atomically rename to the final name when done.
- Rename is atomic on the same filesystem. Copy is not. Always prefer rename.
- Long-running writers keep the .PROCESSING file open for minutes or hours — it is clearly visible.
- One-shot writers create and rename in milliseconds — the pattern works, you just can't see it. Check logs to confirm.
- Java's renameTo() silently returns false on failure — always check the return value or use Files.move() with ATOMIC_MOVE.
- Handle orphan .PROCESSING files — clean them up on failure and on application startup.
- Consumer cooperation is essential — filter by extension, check file stability and move before processing.
- The pattern costs almost nothing — two extra lines of code, zero performance impact and massive reliability gains.
- When in doubt, use the pattern. The cost of not using it is always higher than the cost of using it.
🎯 Final Conclusion
The .PROCESSING file pattern is one of those things in software engineering that sounds almost too simple to matter.
Write to a temp name. Rename when done.
Two lines of code.
But those two lines prevent an entire class of bugs that are notoriously difficult to reproduce, diagnose and fix in production.
Whether your file write takes five minutes or five milliseconds, the protection is the same. Whether your system processes 100 records or 10 million records, the pattern scales. Whether you are writing CSV, JSON, XML or binary data, the approach is identical.
Enterprise batch frameworks, file transfer systems, ETL pipelines and deployment tools have used this pattern for decades. It is battle-tested, production-proven and practically free to implement.
If your system produces files that other systems consume — and you are not already using this pattern — you probably should be.
The next time you write a file that another system will read, ask yourself: what happens if someone reads it right now, before I finish writing? If the answer worries you — the .PROCESSING pattern is your two-line fix.
- Have you ever encountered a bug caused by reading an incomplete file in production? How long did it take to diagnose?
- Does your current project use the .PROCESSING pattern (or a similar approach) — or are you writing directly to final filenames?
- What other file-handling patterns have saved you from production issues?
Drop your thoughts in the comments below 👇
If this helped you understand why safe file writing matters, share it with a developer who might need it.
#SoftwareEngineering #Backend #FileProcessing #BatchProcessing #DesignPatterns #Java #Python #AtomicWrite #DataIntegrity #TechSimplified #ETL #ProductionBugs
Home | Software Engineering | Tech Simplified
Sources and Further Reading: POSIX rename() specification — atomic file rename semantics | Java Documentation — File.renameTo() and java.nio.file.Files.move() | Linux man pages — rename(2) system call behaviour | Spring Batch Documentation — ItemWriter and file handling patterns
Editorial note: File system behaviour can vary across operating systems and configurations. The patterns described here apply to standard POSIX-compliant and Windows NTFS filesystems. Always test on your target environment.
Last reviewed: August 2026
Did you find this post helpful?
Never miss a post!
Get fresh insights delivered to your inbox.
OR
No spam. Unsubscribe anytime.
0 Comments
We’d love to hear your thoughts. Feel free to comment below!