How to test the performance of your SMPP server
Ozeki SMS Gateway is a very fast SMS Gateway system. It can transfer up to 1000 SMS/second on standard server computer. This document show how you can test the SMPP performance of your SMS gateway system.
Ozeki SMS gateway contains built in functionality for SMS performance testing. You can use this functionality to test the performance of your SMPP service. You can test the SMPP client performance or the SMPP server performance independently or you can setup a test system from two computers and test the overall throughput of the system. In this guide you can learn about the testing components, and we present a system you can setup in your own network to test how your services perform.
How to test the performance of your SMPP server (Simple directives)
To test the performance of your SMPP server:
- Launch Ozeki SMS Gateway
- Add new Tester user
- Configure Tester user connection
- Select Connect toolbar icon
- Install Tester connection
- Enter Tester connection name
- Go to the Tester user's Test tab
- Choose performance test
- Check SMPP server performance in the logs
SMPP speed test system architecture
The test system we use for performance testing consists of two computers: Computer "A" and Computer "B". Computer "A" has an SMPP client installed and the tester user component is configured to help us generate test messages. Computer "B" has an SMPP server configured that accepts the SMPP client connection of Computer "A". Computer "B" accepts all SMS messages sent by Computer "A" and forwards these messages to a so-called Tester connection. The tester connection swallows all the messages. This architecture is presented on Figure 1.
How much does the performance test cost?
This testing setup does not forward any messages to the mobile network, so you will not have to worry about SMS charges. All messages stay in house. The test messages are created on Computer "A", they travel through your computer network to Computer "B", where they are "eaten" by the "Tester connection".
What kind of hardware should I use?
You can use any computer hardware that is capable of running Windows or Linux and
that meets the SMS gateway system requirements. Note that
the performance of your system highly depends on the hardware configuration. In the
test we have done, we have used the following setup:
Computer A:
CPU: Intel(R) Core(TM) i7-9700K CPU@3.60 GHz
RAM: Corsair Vengeance 64 GB (2x32GB) 3200Mhz DDR4
HDD: Samsung 970 EVO Plus 1TB NMe M.2(2280) Solid State Drive (SSD)
Motherboard: ASUS Prime Z390M-Plus LGA1151 Gen2
OS: Windows 10 Professional
Software: Ozeki 10 SMS Gateway v10.3.100
Computer B:
CPU: Intel(R) Core(TM) i7-9700K CPU@3.60 GHz
RAM: Corsair Vengeance 64 GB (2x32GB) 3200Mhz DDR4
HDD: Samsung 970 EVO Plus 1TB NMe M.2(2280) Solid State Drive (SSD)
Motherboard: ASUS Prime Z390M-Plus LGA1151 Gen2
OS: Windows 10 Professional
Software: Ozeki 10 SMS Gateway v10.3.100
Network: 1Gbps Ethernet link through an HP desktop switch
This setup was capable of delivering 1000 SMS messages per seconds. It delivered 1 000 000 SMS messages from computer A to computer B in less than 16 minutes.
Configuring Computer "B"
When you start to configure the system, it is a good idea to start with the configuration of computer "B". The reason for this is that this computer provides the SMPP Server functionality, and it is better to create the SMPP user account first and configure the client on Computer "A" later to use this user account.
The first video shows you the steps to take to configure computer "B". The video starts from the default installation. To install the system, please follow the instructions in the Ozeki SMS Gateway installation guide. After the system has been installed, you should setup a tester connection and an SMPP user to accept incoming SMPP client connections. By installing the SMPP user, Ozeki 10 SMS Gateway automatically enables and configures the built in SMPP server. This SMPP server will listen on port 9550. Note that during the SMPP user setup we have provided a username and a password. We have used smp1 as the username, and qwe123 for the password. These credentials will also be used on Computer "A" to configure the SMPP client.
Configuring Computer "A"
Video number two shows the steps to take to configure computer "A". During the configuration, first we create an SMPP client connection. During the SMPP client setup, we provide the IP address of Computer "B", along with the SMPP username and SMPP password. Note that we use the same credentials (smp1, qwe123), that we have configured on Computer "B" when we have created the SMPP user account.
After the SMPP client connection is created we setup a tester user. The test user is a built-in testing functionality that can be used to send test SMS messages through the system. You can generate and send large volumes of messages with a single click. Each message will have its own unique phone number and a unique message text to simulate real time conditions. Once the test user is configured you are ready to perform the test.
Note that we have assumed that the firewall on Computer "A" will allow incoming SMPP (TCP/IP) connections on port 9550. If you have not yet configured this on your firewall, you can find the detailed instructions for firewall setup on the how to configure my firewall for SMPP server page.
To test the performance of your system, you need to generate many SMS messages. This can be done with the help of the test user. To install a tester user, click on the Apps button on the toolbar, and as shown in the next figure, scroll down to the Testing section and click on the Install button of the Tester user connection. (Figure 2)
Before the tester user is created, the configuration form shows up. This connection does not need any further configuration, all you need to do is to simply specify a name for the connection. (Figure 3) If that is done, just click on OK to create the tester user.
The key to creating the test system is the installation of the Tester service provider connection. This connection will be used as a test service provider, where the Tester user connection connects to. To create this connection, select the Connect menu from the toolbar, and from the Connections for testing section. To install this connection, you have to select it from the service provider list, and you need to click on the Install button. (Figure 4)
After you click install, you will have to provide a name for that tester connection, and you can perform further configuration on the advanced tab, such as setting the logging level or enabling delivery reports. (Figure 5)
Performing the test
During the test we have sent 1 000 000 SMS message from computer "A" to computer "B". This event was recorded on both computers. Note that during the test video capture was in progress on both computers which had some effect on the performance. Below you can see the unedited videos showing the message throughput on computer "A" and computer "B". What is worth mentioning, that you will see speed measurement labels publishing the speed in real time for different components. You will see that the speed was about 1500 SMS messages per second. This means that it takes about 10-11 minutes to transfer the 1 000 000 SMS messages from computer "A" to computer "B" through the SMPP link.
At this point, you installed both of the required tester connections and your system is ready to be tested. For that, first, open the Tester user connection in the main menu of the SMS Gateway. Here, in the Test tab of the connection, you can select the amount messages the test is going to send. This allows you to test your system only with one single message or initiate 100 000 messages at once. So, as you can see it in Figure 6, select the test with 100 000 messages by clicking on the 'Send test (100,000)' button.
Send 1 000 000 SMS: Computer "A" view
On computer "A" messages were sent from the tester user to the SMPP client through the routing table, and then to the network. During this process you will see that the speed was around 1500 MPS. What is important is that this speed was sustained during the whole test. On the video you will see that first the 1 000 000 messages were added to the Outbox of the system, and they were sent out as capacity became available. You might notice that the Outbox count was split into two components. You can see a black number and a red number. The red number next to the Outbox label shows that some messages were added to a so called "future delivery outbox". This is required to keep the default outbox size (marked with the black number) below 50000 to make sure performance is kept high. The basic idea behind this split, is that working with smaller data sizes is faster. You will also see, that as messages were moved from the standard outbox, the future delivery outbox was emptied in batches of 10000 messages.
Send 1 000 000 SMS: Computer "B" view
On computer "B" messages came into the SMPP user account. After this they went to the routing table and got to the SMPP tester connection. You will notice that all components were operating at a sustainable speed of about 1500 MPS. We also show the odometer value at the end of the test. You will see that this number is standing at 1000021 messages. This number matches the number you have seen in the odometers at computer "A". This means that actually 1 000 021 messages were sent during the test. The 21 extra messages are not some kind of an error. They are carrying meta information about the test and are helping the developers at Ozeki to have a better understanding of the messages flow. You can simply ignore them.
How can I verify the test results (use log files)
The above test was performed without logging turned on. Logging requires disk I/O which means that computer has to do more things at the same time. With logging turn on the SMS throughput decreases somewhat, but in order to verify our test results, to be sure, that all 1 000 000 SMS messages went from computer "A" to computer "B", we need to turn on logging.
The best place to enable logging is at the SMPP user on computer "B". At this point we can record the SMPP transactions for all incoming SMS messages. In the video below SMPP low level logging is turned on for the SMPP user. This means that every SMPP PDU is logged into a text file during the test. After logging was turned, we have sent a batch of 100 000 SMS messages to the system. You will notice that the speed went down to about 1300 MPS with logging enabled. If You watch the video, you can see that with logging enabled you can monitor the SMPP communication in real time on the GUI. At the end of the video, you will see that about 76 Mbytes of log data was written to the disk. 100 000 SMS resulted 76 Mb of log when we enabled SMPP logging. By reading the log in notepad you can verify that all 100 000 SMS messages were received. If you don't need a detailed log of each SMPP transaction, we recommend that you turn off low level logging to save resources.
Running a performance test with delivery reports
The SMS infrastructure offers delivery reports, which are confirmations returned to the user when an SMS arrives the recipient handset. Delivery reports are carried as standard SMS messages in the mobile network. If a delivery report is requested for, a so-called delivery report SMS is returned to the sender when the SMS arrives to the recipient. This means that every outgoing SMS generates an incoming SMS. If you require delivery reports, you need double capacity from your SMS gateway.
One good thing about delivery reports is that they travel in the opposite direction, so the network is network speed is not halved when delivery reports are enabled.
In the final test we turned on delivery report simulation in the tester connection on computer "B" and we have enabled delivery report requests in the tester user on computer "A". This configuration allows us to test the performance of Ozeki 10 SMS Gateway with delivery reports enabled.
Configuring Computer "A" for Delivery reports
To configure computer "A" to ask for delivery reports for each test SMS, you need to open the Advanced tab of the configuration form of the tester user. In the advanced tab open the Delivery reports Group box and tick the "Request delivery reports" checkbox.
Configuring Computer "B" for Delivery reports
To configure computer "B" you need to perform two tasks. First you should enable delivery report simulation in the tester connection. You can do this by opening the configuration form of the tester connection. On this form select the Advanced tab, open the Delivery reports group box and click the "Return delivery reports" checkbox.
The next step is to configure delivery report support in the SMPP user settings. To do this open the configuration form of the SMPP user and make sure that the cleanup policy is set to "Cleanup on delivery reports", and the delivery report policy is set to "Request if asked by message". The cleanup setting means that the SMS messages sent by the SMPP client will stay in the sent items folder until a matching delivery report comes in. This allows the system to match incoming delivery reports to previously submitted messages. This is a requirement in order to notify the SMPP client connection correctly with the corresponding messages references when the delivery report is returned to them through the SMPP link.
Doing the performance test with SMS delivery reports
In the performance test we did we enabled SMPP low level logging and delivery report matching. You will see that the speed initially jumps up because now we use a two-way transport. And messages can travel in both directions at a rate offered by the protocol implementations and the network. After some time, this speed goes down to about 1000 SMS per second in both directions. The speed fluctuates because the delivery reports are not generated evenly to simulate real life conditions. In real life some delivery reports come in significantly later. (For example, if a mobile phone is switched off and switched back a few days later, the delivery report can come in several days after the original messages was sent.)
You might also notice that the SMPP communication log is updated in real time on the GUI which eats up some capacity. In a real-life system, you will likely have this log disabled.
One more thing worth noticing is the odometer. You will see two numbers increasing instead of one. This is because now we have both outgoing SMS messages and incoming delivery reports. You will also see the inbox queue size increase. This is where delivery reports are stored until they could be transported through the SMPP client link.
At the very end of the test, you will see the speed drop. This is because in this phase no incoming messages need to be processed and the number of delivery reports to be returned is lower than system capacity. If there would be more messages, the speed would jump back up.
Conclusion
We can safely state that Ozeki 10 SMS gateway is able to deliver SMPP SMS messages at a rate of 1000 SMS messages per second through the network. This software is one of the fastest (if not the fastest) in its market segment. You will have a hard time finding something better.
More information