Re: Performance Questions
Re: Performance Questions
- Subject: Re: Performance Questions
- From: Kieran Kelleher <email@hidden>
- Date: Sun, 31 Jul 2011 17:29:51 -0400
IMMEDIATE BAND-AID 60 second FIX
========================
1. Go to Monitor, create a new 'Application' named "Admin" and select the current application executable path, launch an instance and use that Admin instance for sending notifications. (You don't need to copy the app, rename and redeploy to do this as you suggested!)
2. Restart your "Public" instance with properties in WOMonitor to (1) provide multiple OSC's for regular requests using wonder property for the ERXOSC pool, and (2) to turn on concurrency.
3. If your public instance(s) struggle(s) on the next batch of notifications, add more instances.
MEDIUM, LONG TERM IMPROVEMENT
=======================
You only have 76 MB of data, do you expect this to grow significantly in the next year or two?
Your innodb memory allocation is 128MB, so your entire DB fits in memory, albeit there are a few settings that can be tweaked even as it is right now.
For high concurrency on this small dataset, you are better off with a few CPUs and (if necessary to offset costs, less memory than 16GB). Not sure if your server is a physical machine or a VPS that can be reconfigured.
Before I would make recomendations on config settings, you need to decide how much memory you want MySQL to use based on projected peak data size over next 12 months.... 256MB, 512MB, 1GB, 2GB ???
From what I understand, your current issues are related to high concurrency in short periods during and after the times when you send your 20,000 notifications, and IIRC, you have database/woa and apache on one machine with 16GB RAM and 1 CPU
Here is some recommendations to handle that concurrency better. You can do the first 3 right now after the BAND-AID fix above gives you breathing space:
* Turn on concurrency in your app.
* Use ERXObjectStoreCoordinatorPool for handling regular request-response EOF. Start with 3 OSC's per instance and see how it goes.
* Use a background thread for your NotificationsSending process with its own OSC (easy way: just extend ERXAbstractTask, use its newEditingContext() method in your task, or copy the logic in that class to your Runnable background task.)
* If your server is a VPS that can have its configuration changed, consider more CPUs (4?) and less total memory, if that offsets the cost of more CPUs, with memory based on next 12 month peak traffic and data size expectations (4GB?)
Then
* Monitor memory for your app instance(s) and adjust as needed.
* You might find that you handle the peak traffic bursts by having a few instances rather than just one
* Monitor, measure and adjust again.
So just maybe an ideal config for your scenario right now might be sth like this:
MySQL: allocate 512MB, covers 6x data size growth, and edit my.cnf to make the best use of that 512MB.
WOA App Instances: 4-6 instances x ???MB each
Some other observations:
TABLE INDICES
-------------------
* Every index adds time to inserts, updates and adds space to the database size, so no need for redundant indices. For example, the following join table has 3 indices and all (single and compound indices) begin with 'app_dev_Id', so therefore this one, KEY `app_dev_Id` (`app_dev_Id`), is redundant. Either of the other two covers that index requirement.
CREATE TABLE IF NOT EXISTS `app_dev_not_type` (
`app_dev_Id` bigint(20) NOT NULL default '0',
`not_type_id` int(11) NOT NULL default '0',
`active` int(11) default '1',
`create_date` datetime default NULL,
`modify_date` datetime default NULL,
PRIMARY KEY (`app_dev_Id`,`not_type_id`),
KEY `app_dev_Id` (`app_dev_Id`),
KEY `app_dev_Id_3` (`app_dev_Id`,`active`),
KEY `idx_not_type_id` (`not_type_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
is the same as:
CREATE TABLE IF NOT EXISTS `app_dev_not_type` (
`app_dev_Id` bigint(20) NOT NULL default '0',
`not_type_id` int(11) NOT NULL default '0',
`active` int(11) default '1',
`create_date` datetime default NULL,
`modify_date` datetime default NULL,
PRIMARY KEY (`app_dev_Id`,`not_type_id`),
KEY `app_dev_Id_3` (`app_dev_Id`,`active`),
KEY `idx_not_type_id` (`not_type_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
* it is a good practice to add index for the reverse relationship in join tables, for example:
CREATE TABLE IF NOT EXISTS `device_notification` (
`device_id` bigint(20) NOT NULL default '0',
`notification_id` bigint(20) NOT NULL default '0',
PRIMARY KEY (`device_id`,`notification_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
You should put compound index for the reverse relationship, so it should be:
CREATE TABLE IF NOT EXISTS `device_notification` (
`device_id` bigint(20) NOT NULL default '0',
`notification_id` bigint(20) NOT NULL default '0',
PRIMARY KEY (`device_id`,`notification_id`),
KEY `reverse_rel` (`notification_id`,`device_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
SUMMARY
==========
Andrew, it seems your primary issue here is EOF concurrency and app concurrency in general during and immediately after you send your large group of notifications.
HTH, Kieran
On Jul 29, 2011, at 3:38 PM, Andrew Kinnie wrote:
> OK, thanks.
>
> I have implemented a simple task version of my old utilities class to run the sends, but I did not know that about EOF being single threaded or the single db connection. I will look at the ERXTaskObjectStoreCoordinatorPool.
>
> BIG help.
>
> I attached the query results and the show variables result
>
> <query_result.csv><variables.csv><my.cnf><apns schema.sql>
>
> Note, based on Pascal's suggestion we have bumped the memory on MySQL, but it is not reflected yet, and won't be until we restart it at 3:30 AM. Hopefully there won't be any major breaking news at that time.
>
>
>
> On Jul 29, 2011, at 2:56 PM, Kieran Kelleher wrote:
>
>> Andrew,
>>
>> Hi Andrew,
>>
>> Just to endorse what some have said, after reading this thread:
>>
>> 1) concurrency must be ON
>> 2) For your 1 minute task do it in a background thread and use a different OSC. Remember EOF is a single-threaded, single-db-connection stack. If you want high concurrency performance, you cannot just use the default OSC. Use a ERXTaskObjectStoreCoordinatorPool just for tasks, even if it is just a pool of one.
>>
>> Also, if I get a few minutes later or at the weekend, I can eyeball your setup for possible low-hanging fruit if you send the following:
>>
>> A) Send your /etc/my.cnf file to the list, and tell me how much total max memory you want mysql to have - I will take a quick look at it to see if it looks OK.
>>
>> B) send the output of the following SQL statement in a text file:
>> select TABLE_SCHEMA, TABLE_NAME, TABLE_ROWS, (DATA_LENGTH + INDEX_LENGTH)/1024/1024 as SIZE_IN_MB, DATA_LENGTH/1024/1024 as DATA_SIZE_IN_MB, INDEX_LENGTH/1024/1024 as INDEX_SIZE_IN_MB from information_schema.TABLES order by SIZE_IN_MB desc;
>>
>> C) send the output of the following SQL statement in a text file:
>> SHOW VARIABLES;
>>
>> D) Send the output (allschemas.sql) of the following CLI statement:
>> mysqldump --all-databases --opt --no-data > allschemas.sql
>>
>>
>>
>>
>>
>>
>>
>> On Jul 29, 2011, at 10:56 AM, Andrew Kinnie wrote:
>>
>>> That will help, thanks!
>>>
>>> On Jul 29, 2011, at 10:55 AM, Alexis Tual wrote:
>>>
>>>> An example of all that John said is available there thanks to Kieran :
>>>>
>>>> https://github.com/projectwonder/wonder/tree/master/Examples/Misc/BackgroundTasks
>>>>
>>>> Alex
>>>>
>>>> Le 29 juil. 2011 à 16:52, Andrew Kinnie a écrit :
>>>>
>>>>> Thanks. I may give that a try. That was one of the other options I thought of, but was hoping to avoid a significant re-write.
>>>>>
>>>>>
>>>>>
>>>>> On Jul 29, 2011, at 10:44 AM, John & Kim Larson wrote:
>>>>>
>>>>>> rather than increasing worker threads, why not just spawn a new Java thread for sending the notifications? That thread can run in the background while you're doing EO stuff and free your app up to do the servicing of requests.
>>>>>>
>>>>>> If you go down this path, I always pass EOs to other threads as globalIDs to prevent problems. Also, make sure you don't lock the OSC for the app during your work or your app will hang while other threads' ECs wait to get it. If this gets bad enough, use a separate OSC stack and dispose of it when your done.
>>>>>>
>>>>>> John
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Jul 29, 2011, at 9:28 AM, Andrew Kinnie <email@hidden> wrote:
>>>>>>
>>>>>>> Greetings
>>>>>>>
>>>>>>> I have a deployed app which serves as a push notification server for our iOS app. It uses a recent ERRest (post WOWODC) to provide access to the data which is located on a MySQL database (using innoDB). The model has entities for PushApplication (the iOS app), ApplicationDevice (i.e. an iOS device which has our iOS app), Notification and has a lookup table for NotificationType (5 rows). Notification is a message, and there is a many to many with ApplicationDevice along with a corresponding device_notification table, as well as ApplicationDeviceNotificationType to allow particular devices to have particular types of notifications turned on or off.
>>>>>>>
>>>>>>> Our app in connected to by our editorial staff via a Cold Fusion app to send out breaking news alerts as push notifications. I then get (via a fetch) all the devices which have that particular notification type (basically 90% of our 20,000+ "installed" applicationDevices), then I pass that array into a method which makes the connection to Apple and iterates through the array sending one notification to each device in turn, then closes the connection.
>>>>>>>
>>>>>>> It takes approximately 1 minute to send an alert to all 20,000 devices.
>>>>>>>
>>>>>>> While this happens, some of these devices are getting the push from Apple (which is crazy fast about it), and some of them are running the app and the iOS app itself is querying the server for details about the notification and loading it in. However, if this happens while the push is still in the process of sending (i.e. within the 1 minute time frame), the iOS app may be forced to wait for the send process to finish (as many as 60 seconds presumably. It doesn't happen all that often, because our app doesn't buzz or makes a sound when it receives a notification, but it is not ideal. We anticipate using this same app and server for the Android version, and for the upcoming iPhone update, so the number of installed devices could increase pretty dramatically. Currently it is iPad only.
>>>>>>>
>>>>>>> So, we're trying to figure out what to do about it. Currently the app is deployed on a CentOS server (single core processor) which also houses the db, but nothing else. It has 16 GB of RAM.
>>>>>>>
>>>>>>> We were considering:
>>>>>>>
>>>>>>> 1. Trying to increase the threads the app can create, but I'm not sure that would fix it as much as mask it
>>>>>>> 2. Trying to run an additional copy of the app to send while the other one handles the incoming client requests, but I am not sure how to accomplish this other than copying the whole project, renaming it, then deploying that. I am also not sure this would fix anything if in fact the issue were locking in the database or jdbc or something of that nature.
>>>>>>> 3. Seeing if there was something easier, more efficient and less kludgy feeling than either of those. (assuming either of those would work anyway, we have some difficulty testing it without sending out 20,000 push notifications)
>>>>>>>
>>>>>>> Anyone have any insight?
>>>>>>>
>>>>>>> Andrew
>>>>>>> _______________________________________________
>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>
>>>>>>> This email sent to email@hidden
>>>>>
>>>>> _______________________________________________
>>>>> Do not post admin requests to the list. They will be ignored.
>>>>> Webobjects-dev mailing list (email@hidden)
>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>
>>>>> This email sent to email@hidden
>>>>
>>>
>>> _______________________________________________
>>> Do not post admin requests to the list. They will be ignored.
>>> Webobjects-dev mailing list (email@hidden)
>>> Help/Unsubscribe/Update your Subscription:
>>>
>>> This email sent to email@hidden
>>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden