• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: Performance in many objects manipulation
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Performance in many objects manipulation


  • Subject: Re: Performance in many objects manipulation
  • From: Mark Wardle <email@hidden>
  • Date: Thu, 10 Mar 2016 16:05:57 +0000

I don’t usually fetch all objects but fetch in batches.

You need to recycle the EOEditingContext. I tend to bring the array in batches, process, then create a new editing context and repeat.

I have some code included below but I think there is also good batching support in Wonder. The code below fetches raw rows and then converts to an EO when requested. The code below will automatically use a new editing context for each batch.

Mark


package com.eldrix.snomedct.search;

import java.util.Iterator;

import org.apache.log4j.Logger;

import com.webobjects.eoaccess.EODatabaseContext;
import com.webobjects.eoaccess.EOEntity;
import com.webobjects.eoaccess.EOSQLExpression;
import com.webobjects.eocontrol.EOEditingContext;
import com.webobjects.eocontrol.EOFetchSpecification;
import com.webobjects.eocontrol.EOSortOrdering;
import com.webobjects.foundation.NSArray;
import com.webobjects.foundation.NSDictionary;

import er.extensions.eof.ERXEC;
import er.extensions.eof.ERXEOAccessUtilities;
import er.extensions.eof.ERXEOControlUtilities;

/**
 * This is an optimised memory-efficient method of iterating through enterprise objects.
 * This will iterate through enterprise objects, either returning the primary key, the object or the raw row.
 *
 * For example:
 * <code>
 * <pre>
 * for(EOEnterpriseObject eo: IterableBatchEnterpriseObjects.allObjectsOfEntity("Patient", 300) {
 * 		// process eo
 * }
 * </pre>
 * </code>
 *
 * @author mark
 *
 */
@SuppressWarnings("rawtypes")
public class IterableBatchEnterpriseObjects implements Iterable<Object> {
	private static Logger log = Logger.getLogger(IterableBatchEnterpriseObjects.class);
	private final static int DEFAULT_BATCH_SIZE=50;
	private EOEditingContext _ec;
	private EOFetchSpecification _fs;
	private int _rowCount;
	private int _start;
	private int _batchSize;
	private NSArray _currentBatch;
	private int _batchPosition;
	private IterableBatchMode _mode;
	private boolean _refreshEditingContext = true;		// we create a new editing context for each batch by default

	private enum IterableBatchMode {
		PRIMARY_KEYS {
			 NSArray resultsInRange(EOEditingContext ec, EOFetchSpecification fs, int start, int end) {
				 return ERXEOControlUtilities.primaryKeyValuesInRange(ec, fs, start, end);
			 }
		},
		OBJECTS {
			NSArray resultsInRange(EOEditingContext ec, EOFetchSpecification fs, int start, int end) {
				return ERXEOControlUtilities.objectsInRange(ec, fs, start, end);
			}
		},
		RAW_ROWS {
			NSArray resultsInRange(EOEditingContext ec, EOFetchSpecification fs, int start, int end) {
				return rawRowsInRange(ec, fs, start, end);
			}
		};

		abstract NSArray resultsInRange(EOEditingContext ec, EOFetchSpecification fs, int start, int end);
	};


	private IterableBatchEnterpriseObjects(EOFetchSpecification fs, int batchSize, IterableBatchMode mode) {
		_fs = fs;
		_batchSize = batchSize;
		_mode = mode;
		_rowCount = ERXEOAccessUtilities.rowCountForFetchSpecification(editingContext(), _fs);
	}

	public static IterableBatchEnterpriseObjects allObjectsOfEntity(String entityName, NSArray<EOSortOrdering> sortOrderings, int batchSize) {
		EOFetchSpecification fs = new EOFetchSpecification(entityName, null, sortOrderings);
		return new IterableBatchEnterpriseObjects(fs, batchSize, IterableBatchMode.OBJECTS);
	}
	public static IterableBatchEnterpriseObjects allObjectsOfEntity(String entityName, int batchSize) {
		return allObjectsOfEntity(entityName, sortByPrimaryKey(entityName), batchSize);
	}
	public static IterableBatchEnterpriseObjects allObjectsOfEntity(String entityName) {
		return allObjectsOfEntity(entityName, DEFAULT_BATCH_SIZE);
	}

	public static IterableBatchEnterpriseObjects allPrimaryKeysOfEntity(String entityName, NSArray<EOSortOrdering> sortOrderings, int batchSize) {
		EOFetchSpecification fs = new EOFetchSpecification(entityName, null, sortOrderings);
		return new IterableBatchEnterpriseObjects(fs, batchSize, IterableBatchMode.PRIMARY_KEYS);
	}
	public static IterableBatchEnterpriseObjects allPrimaryKeysOfEntity(String entityName, int batchSize) {
		return allPrimaryKeysOfEntity(entityName, sortByPrimaryKey(entityName), batchSize);
	}
	public static IterableBatchEnterpriseObjects allPrimaryKeysOfEntity(String entityName) {
		return allPrimaryKeysOfEntity(entityName, DEFAULT_BATCH_SIZE);
	}

	public static IterableBatchEnterpriseObjects objects(EOFetchSpecification fs, int batchSize) {
		return new IterableBatchEnterpriseObjects(fs, batchSize, IterableBatchMode.OBJECTS);
	}

	public static IterableBatchEnterpriseObjects primaryKeys(EOFetchSpecification fs, int batchSize) {
		return new IterableBatchEnterpriseObjects(fs, batchSize, IterableBatchMode.PRIMARY_KEYS);
	}

	public static IterableBatchEnterpriseObjects rawRows(EOFetchSpecification fs, int batchSize) {
		return new IterableBatchEnterpriseObjects(fs, batchSize, IterableBatchMode.RAW_ROWS);
	}

	public static NSArray<EOSortOrdering> sortByPrimaryKey(String entityName) {
		EOEntity entity = ERXEOAccessUtilities.entityNamed(null, entityName);
		NSArray<String> attrs = entity.primaryKeyAttributeNames();
		EOSortOrdering so = EOSortOrdering.sortOrderingWithKey(attrs.get(0), EOSortOrdering.CompareAscending);
		return new NSArray<EOSortOrdering>(so);
	}

	/**
	 * Returns an {@link com.webobjects.foundation.NSArray NSArray} containing the raw rows for the fetch specification.
	 * This is based upon ERXEOControlUtilities.primaryKeyValuesInRange
	 * @param ec
	 * @param spec
	 * @param start
	 * @param end
	 * @see er.extensions.eof.ERXEOControlUtilities#primaryKeyValuesInRange
	 * @return
	 */
	public static NSArray rawRowsInRange(EOEditingContext ec, EOFetchSpecification spec, int start, int end) {
        EOEntity entity = ERXEOAccessUtilities.entityNamed(ec, spec.entityName());
        NSArray attributeNames = (NSArray) entity.attributes().valueForKey("name");
		spec.setFetchesRawRows(true);
		spec.setRawRowKeyPaths(attributeNames);
    	EOFetchSpecification clonedFetchSpec = (EOFetchSpecification)spec.clone();
        EOSQLExpression sql = ERXEOAccessUtilities.sqlExpressionForFetchSpecification(ec, clonedFetchSpec, start, end);
        NSDictionary<String, EOSQLExpression> hints = new NSDictionary<String, EOSQLExpression>(sql, EODatabaseContext.CustomQueryExpressionHintKey);
        clonedFetchSpec.setHints(hints);
        return ec.objectsWithFetchSpecification(clonedFetchSpec);
	}


	public EOEditingContext editingContext() {
		if (_ec == null) {
			_ec = ERXEC.newEditingContext();
		}
		return _ec;
	}
	public void setEditingContext(EOEditingContext ec) {
		_ec = ec;
		_refreshEditingContext = false;
	}

	protected EOFetchSpecification fetchSpecification() {
		return _fs;
	}

	protected NSArray currentBatch() {
		if (_currentBatch == null) {
			if (_start > _rowCount) return null;
			int end = _start + _batchSize;
			if (end > _rowCount) end = _rowCount;
			log.debug("Fetching batch... start: " + _start + " end: " + end);
			_currentBatch = _mode.resultsInRange(editingContext(), fetchSpecification(), _start, end);
			_batchPosition = 0;
		}
		return _currentBatch;
	}

	/**
	 * Fetch next batch, optionally creating a new editingcontext for this new batch
	 * @return
	 */
	protected NSArray nextBatch() {
		_start += _batchSize;
		_currentBatch = null;
		if (refreshEditingContext()==true) _ec = null;
		return currentBatch();
	}

	public boolean hasNext() {
		if (_start + _batchPosition >= _rowCount) return false;
		return true;
	}

	public Object next() {
		Object eo = null;
		NSArray batch = currentBatch();
		if (_batchPosition == batch.size()) {
			batch = nextBatch();
		}
		if (batch != null) {
			eo = batch.get(_batchPosition);
			_batchPosition++ ;
		}
		return eo;
	}


	public void setRefreshEditingContext(boolean refresh) {
		_refreshEditingContext = refresh;
	}
	public boolean refreshEditingContext() {
		return _refreshEditingContext;
	}

	protected class EnterpriseObjectPrimaryKeyIterator implements Iterator {

		public boolean hasNext() {
			return IterableBatchEnterpriseObjects.this.hasNext();
		}

		public Object next() {
			return IterableBatchEnterpriseObjects.this.next();
		}

		public void remove() {
			throw new UnsupportedOperationException();
		}
	}

	public Iterator<Object> iterator() {
		return new EnterpriseObjectPrimaryKeyIterator();
	}
}


> On 10 Mar 2016, at 10:51, Daniele Corti <email@hidden> wrote:
>
> Hi,
> I have a question, about performance while working with LongRequest and many EO.
>
> Here's the situation: I've prepared a class that extends ERXLongResponseTask.DefaultImplementation.
> In this class, I operate a database migration, over 10000 record.
> Each of them has, at least, 50/60 objects releated. From them, I create others objects, for a total of 60 new objects for each of the starting 10000 rows.
>
> The procedure works fine, but, I experience a difference in speed during the procedure. At first, 300-400 objects are processed in few seconds, then the procedure slow down until, from 5000, it processes about 7/8 objects in 5 seconds.
>
> The procedure is very simple:
> 1. Fetch all objects in a NSArray
> 2. Manipulate each object in a for loop, fetching related objects and creating other objects in every iteration.
>
> I use just one EOEditingContext, so, I think the slowness begin when it became really full of elements.
>
> I would like to know, if there is a way to "clean up" the manipulated objects from the EC, or if a different approach is preferred.
>
> One last thing: it is good to use a NSArray of 1000 object in the for loop to do this? I ask this because, I fetch all the objects and save them in memory using EOObject.fetchAllObjects(EC) method from the class.
>
> Thanks in advance!
> Daniele C.
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list      (email@hidden)
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden


 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden


References: 
 >Performance in many objects manipulation (From: Daniele Corti <email@hidden>)

  • Prev by Date: Performance in many objects manipulation
  • Next by Date: Re: Performance in many objects manipulation
  • Previous by thread: Performance in many objects manipulation
  • Next by thread: Re: Performance in many objects manipulation
  • Index(es):
    • Date
    • Thread