Import files from a directory and save them to AWS S3 + Database
Recently I needed to take some files within a folder and copy them up to our cloud storage (AWS S3), whilst also saving the file reference to a database.
It was more of an ad-hoc task, which is why i decided to use an artisan command to help achieve it. Artisan commands come in really handy if you need to create a script which you can run as-and-when or automate using a cron job.
In this post I'll cover :
- Reading files from a directory
- Extracting the filenames
- Import each file to cloud storage (AWS S3)
- Saving the file reference and name to a database
Let's begin by creating our new artisan command: php artisan make:command ImportDocuments
This will create a new file in app/Console/Commands. Open that up and let's continue.
The first part of the file, will be the name and signature of the command. This will be what you type in on the terminal within your project to run the command. It can also accept other arguments which i'll talk about next. For the mean time, change your code to look like so:
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'import:documents
{importPath : The path to the directory which contains the files.}
{--dry-run : Only validate that the files can be read. No uploads or saving.}';As you can see we have additional paramaters in there which are called input descriptions, you can add as many as you need and they work as follows: php artisan import:documents ~/path/to/folder --dry-run. For each input you expect, just add a space after everyone when typing out the command.
Next we'll move onto the handle() method and write the bulk of our import script.
/**
* Execute the console command.
*
* @return int
*/
public function handle(): int
{
// Fetch the inputs from the command line
$importPath = $this->argument('importPath');
$isDryRun = $this->option('dry-run');
// Output some feedback to the user
$this->info('Reading files in directory...');
// Create an array of pathnames which match a given pattern
// Change the file extension to match the files you plan to import
$files = glob("{$importPath}/*.pdf");
// Check if we found any files
if (!$files) {
$this->error('Unable to read any files in the directory.');
return 1;
}
// Create a progress bar to let the user know how many files have been imported
$progress = $this->output->createProgressBar(count($files));
$progress->start();
// Loop over each file and import it
foreach ($files as $file) {
// Optional - extract an ID from the file name to assign the file to a user in your system
// E.g. /path/to/files/example-file-123456.pdf
// Extract the user id from the filename - the user_id is 6 digits
preg_match('/[0-9]{6}/', $file, $matches);
$userId = $matches[0];
// Attempt to find the user or Bail out
$user = User::findOrFail($userId);
// Strip out the full file path, but keep the original file name
$newFileName = str_replace($importPath . '/', '', $file);
// If this is a dry run, stop here and don't process any files
if (!$isDryRun) {
// Read in the file
$pdfFile = fopen($file, 'r');
// Set the new file path that you want to store in S3
$filePath = strtolower('/' . config('app.env') . '/imports/' . $newFileName);
// Copy over the document to S3, replace private, with public if needed
Storage::disk('s3')->put($filePath, $pdfFile, 'private');
// Optional - Store the file reference in the database
// Replace with your own model and fields
Document::create([
'user_id' => $user->id,
'name' => 'Imported File',
'file_path' => $filePath
]);
fclose($pdfFile);
}
// Advance the progress bar by 1
$progress->advance();
}
$progress->finish();
return 0;
}I've commented my code to help explain each step. There are a few niceties such as the progress bar and the info text. These are optional, take them out if you want.
If you want to know more about artisan commands, take a look at the official documentation.