This article is a case study for integrating Magento with other systems when web services are not suitable, and we will go through some real life examples.
By default, Magento implements two types of web services: SOAP and REST. This does not require that you know how Magento is built, how it works internally. You can perform almost any operation that Magento can do by default (create orders, add customers, get list of products, etc.) just by consuming web services.
Some examples of usage:
- you need to get the list of orders in your ERP software;
- your courier assigns AWB to orders and marks the orders in Magento when they are shipped;
- you want to show a few products on your blog;
However, this approach is not always suitable, because things can get messy. You may need an intermediary software to achieve the results you want.
Note: in this case the term “Middleware” does not refer to the design pattern used in microframeworks (although this could help in the implementation), it refers to a piece of software that stands between two systems helping them communicate.
Case study #1: Multi Store Stock Middleware
I once worked on a project which had, as part of its multichannel selling strategy, integrated a chain store and an online shop. Nothing fancy, the usual business scheme for this context. However, the challenge came with the request that the stock of the products (common for all the selling points) had to be updated in real time in Magento and consequently shown on the storefront.
The beauty was that the chain store did not have a dedicated warehouse. They had approximately 100 physical stores and each of these stores shared the stock with the online store (half / half). Each time products were sold in the physical store, the stock needed to be updated in Magento. This could have turned messy if all 100 stores started to update the stock at the same time, with the Magento API.
So we built a Middleware which centralizes the data and updates the stock in Magento (SOAP) in a transaction safe manner. Below you can see a simplified version of the Middleware’s flow diagram.
Some notes about the Middleware we created in this case:
- it can be accessed by web services (SOAP, REST, JSONRPC);
- it had access to other systems also by web services;
- it had its own database;
- it is the owner of the stock - that’s why when changes happened, the Middleware needed to be updated.
When you should use this approach:
- when there are multiple data sources, and data need to be centralized before storing it in Magento;
- when logic can be set apart from Magento;
- when the load would be high on Magento, in which case the Middleware can work on a separate environment without affecting the load on the Magento server.
When you should NOT use this approach:
- when you can solve the problem just by using Magento web services (there is no concurrency between client sides).
So how was this application built?
I used Zend Framework 2 for implementing this middleware along with Doctrine DBAL. I implemented a SOAP server in Zend, which is similar to Magento’s SOAP implementation: it is WSI compliant, and you need to have a valid session ID to access any of the methods. You can get the session ID by accessing the “login” method of the SOAP server, where you need to provide a valid user and password.
The stores (ERP) are accessing this SOAP interface whenever a stock update is needed. An SKU of a product, and a number - telling how many items should the Middleware add to the stock - is sent to the middleware. This number can be positive (if new products have been added to the store) or negative (if products have been sold in the store). The ERP is quite limited, it doesn’t allow many things when it comes to updating products. At first, I wanted to put all incoming requests in a queue, update each of them and notify the ERP. This unfortunately was not possible, I had to send a success or failure message on each request. So as a response, I return a success or failure message on every call, depending on the update status. If the response was a failure, the ERP tries the update later.
On the Middleware side, I store the stock in a MySql server, using the INNODB storage engine for the stock database table. This is essential, because it supports transactions and row level locking. So whenever a request comes, the row of the SKU is locked, so no other request can update it, and this lock is kept until the stock is successfully updated in Middleware and Magento. Since on Magento only the stock is updated and all caching is enabled for WSDL, this works quite fast, under a second. In case two requests are trying to update the same SKU product, the first one locks the row and the second, if it cannot update the stock, it waits a few microseconds, and then retries the update. It keeps doing this for a limited period and if it still cannot update the stock, it returns an error message to the ERP system. This kind of fails are quite rare, they occur sometimes during mass updates.
Obviously, this application is more complex, but I will not get into further details, because this is enough to understand the big picture.
Case study #2: Commissioning system
The second time I put this same decoupling principle into practice was for a commissioning application for an MLM (multi-level marketing) business. With a lot of work and creativity, Magento has been customised so that it could handle the extremely complex MLM business model, which had some quite difficult formulas and rules to assign commissions and bonuses for the users on all levels of the MLM hierarchy tree. Because these calculations were very time consuming, but could be totally set apart from the online shop, we’ve decided to build a separate application for this, whose purpose was to fetch data from Magento, make the calculations and send results back to Magento. This application was able to run on a separate environment, with optimized settings and without affecting Magento’s flows and logic in any negative way.
This is how the high level flow diagram looked like:
When you should use this approach:
- when the business logic is complicated;
- when the logic is not directly related to Magento;
- when the load would be high on Magento - the Middleware could work on a separate environment without affecting the load on the Magento server.
When you should NOT use this approach:
- for small tasks.
Some Technical details Implementing Middleware (in PHP)
- if you are using PHP to implement the Middleware, it would be ideal to use a micro framework (Zend Expressive, Silex, Slim) because you don’t always need the complexity of an MVC framework, and they have well implemented routing systems that will make your webservice implementation easier;
- in case the load is very high and many concurrent clients try to access the Middleware at the same time, you can use a queue system to intercept and distribute the requests (Ex.: Rabbit MQ, Active MQ);
- it’s highly recommended to use a RDBMS that support transactions (MySql, PostgreSql, etc.);
- the Middlewares I worked on were written in PHP and used Doctrine for data manipulation. At first, the implementation was based on Doctrine’s ORM, but soon we gave it up, due to performance issues (it added some extra time in the execution), and we relied only on Doctrine Database Abstraction Layer - which worked perfectly fine.
Using SQL views
Besides Middleware there are other alternatives for integrating Magento with third-parties. There are cases when Magento web services can be quite slow, especially when we want to transfer large amounts of data.
Case study #3: Reporting
For example, we had to create some Business Intelligence reports for a project and the software that we were using for data centralization did not support the complex SOAP implementation that Magento has. On top of that, performance was also an issue. So we decided to use MySql views. Mind you, this requires a bit of understanding of how Magneto stores data, and some advanced query writing skills. But done properly, views can make miracles in terms of speed.
Example of fetching customer related information from attributes without hardcoding the attribute IDs:
CREATE VIEW `customer_attribute` AS SELECT ce.entity_id, ea.attribute_id, ea.attribute_code as attribute, CASE ea.backend_type WHEN 'varchar' THEN ce_varchar.value WHEN 'int' THEN ce_int.value WHEN 'text' THEN ce_text.value WHEN 'decimal' THEN ce_decimal.value WHEN 'datetime' THEN ce_datetime.value ELSE NULL END AS value FROM customer_entity AS ce LEFT JOIN eav_attribute AS ea ON ce.entity_type_id = ea.entity_type_id LEFT JOIN customer_entity_varchar AS ce_varchar ON ce.entity_id = ce_varchar.entity_id AND ea.attribute_id = ce_varchar.attribute_id AND ea.backend_type = 'varchar' LEFT JOIN customer_entity_int AS ce_int ON ce.entity_id = ce_int.entity_id AND ea.attribute_id = ce_int.attribute_id AND ea.backend_type = 'int' LEFT JOIN customer_entity_text AS ce_text ON ce.entity_id = ce_text.entity_id AND ea.attribute_id = ce_text.attribute_id AND ea.backend_type = 'text' LEFT JOIN customer_entity_decimal AS ce_decimal ON ce.entity_id = ce_decimal.entity_id AND ea.attribute_id = ce_decimal.attribute_id AND ea.backend_type = 'decimal' LEFT JOIN customer_entity_datetime AS ce_datetime ON ce.entity_id = ce_datetime.entity_id AND ea.attribute_id = ce_datetime.attribute_id AND ea.backend_type = 'datetime' where attribute_code in ('firstname' , 'lastname'); -- more attribute names can be added here -------------------------------------------------------------------- CREATE VIEW `customer_dimension` AS SELECT ce.entity_id, ce.email, ca_firstname.value as firstname, ca_lastname.value as lastname, ce.created_at, ce.updated_at FROM customer_entity AS ce LEFT JOIN customer_attribute AS ca_firstname ON ca_firstname.entity_id = ce.entity_id LEFT JOIN customer_attribute AS ca_lastname ON ca_lastname.entity_id = ce.entity_id where ca_firstname.attribute = "firstname" AND ca_lastname.attribute = "lastname";
You should use this approach:
- just for reading data (set MySql user permissions accordingly);
- when time is crucial;
- when you have to transfer large amounts of data.
You should NOT use this approach:
- for inserting or updating data;
- if you don’t know how Magento works internally.
Sometimes you have to add another layer or remove the layers you don’t need, when the tools Magento offers do not meet your needs. When you have a business logic that can be set apart from Magento, it is a good idea to do so - web services will make your life easier in this regard.