As a commerce agency, we understand that timeouts and memory issues are common challenges for Magento store owners during bulk processing. Your staff may often experience frustrations with tasks like product updates or data synchronization with external systems, which can result in lost productivity. These problems not only impact your team but also disrupt the smooth operation of your eCommerce platform.
Why Do Timeouts Happen?
In Magento, bulk actions like updating products or synchronizing data with external systems are resource-intensive. These tasks, when performed directly through the admin panel, can easily cause timeouts, especially when handling large datasets or slow external APIs. When resources are tied up in this way, the store’s frontend can experience downtime, leading to a poor user experience and reduced revenue.
How Can Queues Help?
Queues offer a simple but powerful solution. Instead of processing these resource-heavy tasks immediately, they can be placed in a queue to be executed in the background. This method reduces strain on your system, helps avoid timeouts, and ensures that your frontend remains unaffected. By offloading tasks to a queue, you’re allowing the system to process data in a more efficient and controlled manner.
Sample Queue-Based Data Synchronization Implementation
Below is an example of how a queue-based system can be implemented for bulk data synchronization in an eCommerce platform. This approach will help you process large amounts of data without risking timeouts or impacting store performance.
1.
Service Class
The Service Class is responsible for handling the data synchronization process with external systems. It makes sure that data is sent asynchronously to avoid delays or timeouts.
<?php
declare(strict_types=1);
namespace NeoSolax\Service;
use Magento\Framework\HTTP\LaminasClient;
use Psr\Log\LoggerInterface;
class DataSyncService
{
public function __construct(
private LaminasClient $httpClient,
private LoggerInterface $logger
) {}
public function syncData(array $dataItems, string $secretKey, string $baseUrl, string $locale): bool
{
try {
$formattedData = array_map(fn($item) => ['identifier' => $item], $dataItems);
$data = ['data_items' => $formattedData];
$this->httpClient->setUri($baseUrl);
$this->httpClient->setMethod(LaminasClient::POST);
$this->httpClient->setHeaders([
'Content-Type' => 'application/json',
'Authorization' => $secretKey,
'locale' => $locale
]);
$this->httpClient->setRawBody(json_encode($data));
$response = $this->httpClient->send();
return $response->isSuccess();
} catch (\Exception $e) {
$this->logger->error('Error during API communication: ' . $e->getMessage());
return false;
}
}
}
2.
Queue Handler Class
The Queue Handler Class pulls tasks from the queue and sends the data to external systems. This ensures that even large sets of data can be processed without overloading the system.
<?php
declare(strict_types=1);
namespace NeoSolax\Queue;
use NeoSolax\Service\DataSyncService;
use Psr\Log\LoggerInterface;
class SyncHandler
{
public function __construct(
private DataSyncService $syncService,
private LoggerInterface $logger
) {}
public function execute(array $data)
{
try {
$success = $this->syncService->syncData(
$data['items'],
$data['secret_key'],
$data['base_url'],
$data['locale']
);
if (!$success) {
$this->logger->error('Data sync failed.');
}
} catch (\Exception $e) {
$this->logger->error('Error during sync execution: ' . $e->getMessage());
}
}
}
3.
Controller Class
The Controller Class initiates the queue job. It retrieves the data to be synchronized and pushes it into the queue, allowing it to be processed asynchronously without affecting the store’s performance.
<?php
declare(strict_types=1);
namespace NeoSolax\Controller\Adminhtml\Sync;
use Magento\Backend\App\Action\Context;
use Magento\Catalog\Model\ResourceModel\Product\CollectionFactory;
use Magento\Framework\App\Config\ScopeConfigInterface;
use Magento\Framework\MessageQueue\PublisherInterface;
use Psr\Log\LoggerInterface;
class DataReSync extends \Magento\Backend\App\Action
{
public function __construct(
Context $context,
private CollectionFactory $collectionFactory,
private ScopeConfigInterface $scopeConfig,
private PublisherInterface $publisher,
private LoggerInterface $logger
) {
parent::__construct($context);
}
public function execute()
{
try {
$collection = $this->collectionFactory->create();
$items = array_column($collection->toArray(), 'sku');
$messageData = [
'items' => $items,
'secret_key' => $this->getSecretKey(),
'base_url' => $this->getBaseUrl(),
'locale' => 'default_locale'
];
$this->publisher->publish('sync_queue', json_encode($messageData));
$this->messageManager->addSuccessMessage(__('Data re-sync initiated.'));
} catch (\Exception $e) {
$this->logger->error($e->getMessage());
$this->messageManager->addErrorMessage(__('Error occurred during sync initiation.'));
}
}
private function getBaseUrl(): string
{
return $this->scopeConfig->getValue('your/config/base_url');
}
private function getSecretKey(): string
{
return $this->scopeConfig->getValue('your/config/secret_key');
}
}
Necessary XML Configuration Files
To ensure your queue-based system works smoothly, you need to configure certain XML files in your Magento 2 module.
1.
queue.xml (Queue Configuration)
Location: app/code/NeoSolax/YourModule/etc/queue.xml
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/queue.xsd">
<queue name="sync_queue" consumerInstance="NeoSolax\Queue\SyncHandler" />
</config>
This file defines the queue and specifies the consumer that handles messages from the queue.
2.
consumers.xml (Consumer Configuration)
Location: app/code/NeoSolax/YourModule/etc/consumers.xml
<?xml version="1.0"?>
<consumers xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/consumers.xsd">
<consumer name="sync_queue_consumer" queue="sync_queue" handler="NeoSolax\Queue\SyncHandler" />
</consumers>
This file registers the consumer that listens to the queue and processes messages.
3.
communication.xml (Publisher Configuration)
Location: app/code/NeoSolax/YourModule/etc/communication.xml
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/communication.xsd">
<topic name="sync_queue" request="NeoSolax\Queue\SyncMessage" response="NeoSolax\Queue\SyncResponse"/>
</config>
This file defines the publisher and the message formats (request and response) for the queue.
4.
di.xml (Dependency Injection Configuration)
Location: app/code/NeoSolax/YourModule/etc/di.xml
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd">
<type name="Magento\Framework\MessageQueue\PublisherInterface">
<arguments>
<argument name="topics" xsi:type="array">
<item name="sync_queue" xsi:type="object">Magento\Framework\MessageQueue\Publisher</item>
</argument>
</arguments>
</type>
<type name="NeoSolax\Queue\SyncHandler">
<arguments>
<argument name="syncService" xsi:type="object">NeoSolax\Service\DataSyncService</argument>
<argument name="logger" xsi:type="object">Psr\Log\LoggerInterface</argument>
</arguments>
</type>
</config>
This file connects the necessary services and handlers to Magento’s DI (Dependency Injection) system.
5.
Optional crontab.xml (Cron Jobs for Queue Processing)
Location: app/code/NeoSolax/YourModule/etc/crontab.xml
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:App/etc/crontab.xsd">
<group id="default">
<job name="process_sync_queue" instance="Magento\Framework\MessageQueue\Consumer" method="process">
<schedule>* * * * *</schedule>
<config_path>queue/consumers_wait_for_messages</config_path>
<arguments>
<argument name="consumerName" xsi:type="string">sync_queue_consumer</argument>
</arguments>
</job>
</group>
</config>
- This cron job is configured to run every minute (* * * * *), ensuring that the queue consumer is continually processing messages.
- The consumerName should match the consumer defined in consumers.xml.
6.
env.php (Optional Configuration for Message Queue Systems like RabbitMQ)
In case you’re using an external message broker like RabbitMQ, you will need to configure env.php for your queue.
Location: app/etc/env.php
<?php
return [
'queue' => [
'consumers_wait_for_messages' => 1, // Ensures consumers wait for new messages before timing out
'amqp' => [
'host' => 'rabbitmq', // Your RabbitMQ host
'port' => '5672', // Default RabbitMQ port
'user' => 'guest', // RabbitMQ user
'password' => 'guest', // RabbitMQ password
'virtualhost' => '/' // RabbitMQ virtual host
]
]
];
- This file defines how your Magento installation interacts with a message broker like RabbitMQ.
- The queue configuration ensures that your consumers are ready to process messages as soon as they arrive.
Conclusion
By integrating a queue-based system with Magento, you can efficiently handle bulk processing tasks, such as data synchronization or product updates, without overloading the system and causing timeouts or memory outages. The use of queues allows these tasks to be processed in the background, ensuring that your store’s frontend performance remains unaffected. To fully set up a queue system, the following elements are necessary:
- Service and handler classes to process tasks asynchronously.
- Queue configuration in queue.xml and consumers.xml to define the queue and its consumers.
- A publisher configuration in communication.xml to manage message sending.
- Dependency injection configurations in di.xml to ensure that the appropriate services are connected.
- Optional crontab.xml and env.php files for managing asynchronous task execution and message brokers like RabbitMQ.
With this system in place, Magento 2 becomes more scalable and efficient, ensuring smooth operation even during intensive bulk processing tasks.
Leave A Comment