Re: Performance issues with large binary data in database
Re: Performance issues with large binary data in database
- Subject: Re: Performance issues with large binary data in database
- From: Jérémy DE ROYER via Webobjects-dev <email@hidden>
- Date: Wed, 11 Sep 2019 09:42:13 +0000
- Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ingencys.net; dmarc=pass action=none header.from=ingencys.net; dkim=pass header.d=ingencys.net; arc=none
- Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=npdT3XTJd/qtRVmqUouxyXgvMJ1+wYTjS3h6hDwTqII=; b=Pcfs1/DmPyQs1RzXognOttRL4eS3ka0e3WkFxydyci0mWojwit7jDCNR0bPxCWy/zPKdtVlmRj3QX+YTo+rJabdf3V8gol4L2DW06cLy9O8SMbPXxlGqqzW1ZPmGpASH6+zqZr637qyQPbHs3yPi/7gZdB2OKnktDp/ibPPyP/DQ3V1pPUHRqP7ghXLy2JAL7hMr6O2pSVOprQYvKlAHjBly0noTBu69DDPOAAz2aLDT5Rv6mgx+uQfLhQuKrntr548iI7GWZSvM/ajLFZ0KL+vqIJE7IQ8la8WPZPmh11kPsF+a+KUPJlflpnTDJeh3W+Yxu80WXE8BIfNx4nY+7w==
- Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Rtn1vmS4YqoZhkUtysuUXd19ejNltEOunFuuD3C4z6L4aT5iGJkvrFEEJb1Og2PRbB/8SHi9ZZ/pH0/jTlPWU6LQBvXd/FWuA5GkFyV6ludgiN/T6AcTh5J2memkCng8veKzbRbbkHjHlTTKnkuuK5TLYSpxut5cQaLZuDK26HD6BfXETeB/EfTCNWI1qeL5R/nh5zg+9Q3Gs1Pv+ZdpTxrF1Xlg/qlU2nHzjUhkSPBUQkDyG6L80PHlBFxR9ROwl1tyhFx2EILuE6SwBj8L6CRqzFm0DfqBGMcY+VjC6tx0U/Uz3u+FKqo/eckoaPoZGuXkYex0VyYeQwyiU4RxAg==
- Thread-topic: Performance issues with large binary data in database
Hi Markus,
We do store the files on disk to avoid such issue (;-)
When dealing with such problems I do enable debug on adaptor setting
-EOAdaptorDebugEnabled to true.
You will see clearer what does EOF do.
Hope this helps,
Jérémy
Le 11 sept. 2019 à 11:32, Hugi Thordarson via Webobjects-dev
<email@hidden<mailto:email@hidden>> a écrit
:
Hi Markus,
when I had problems like this it was usually because I was "accidentally"
fetching a lot of the blob-containing objects (like EOF firing a relationship
to check delete rules when performing a delete).
But I'd start by attaching a profiler like VisualVM to your application to see
where processing power/memory is being consumed. If that doesn't reveal
anything interesting, you might try unmodeling the BLOB attribute and either
(a) create a cover method on your entity class that just does an SQL query for
the blob data or
(b) create a separate entity for the same table that only contains the BLOB
attribute, and some identifying attributes you can use to fetch the data (no
relationships).
Shots in the dark though :). If you have control over the DB schema I'd move
the blob field to a separate table (or move the data to the filesystem, but
sounds like that's out of the question).
Cheers,
- hugi
On 11 Sep 2019, at 09:12, Markus Ruggiero via Webobjects-dev
<email@hidden<mailto:email@hidden>> wrote:
A customer of mine has issues with performance. We have identified that
ec.saveChanges() sometimes takes ages to finish (tens of seconds, sometimes a
minute or more). The application is basically a document hub, nothing too
fancy. But the developer (who is not available anymore) once decided to store
all documents in the database (PostgreSQL). The docs are primarily PDFs with
sizes of several 100k up to multi mega bytes. The problems cannot be tied
directly to the size of the documents, sometimes a large doc goes through in a
couple seconds whereas sometimes a smaller one takes ages. We have found that
the bottelneck must be inside the editing context, the database statements go
through nicely. I think performance gets lost during ec snapshot handling in
conjunction with JVM memory requirements and garbage collection, but we have
not been able to really pinpoint the location where time is iost.
I tend to recommend to the customer to store the files on disk and only keep
metadata in the database relieving the editing context from handling multi mega
byte snapshots.
So my questions to the community are as follows:
- anyone has experience storing multi-megabyte binary data in the database?
- how would one analyse such a situation (where EOEditingContext et.al is not
debuggable due to lack of source)?
- what would you recommend?
Thanks for any tips.
---markus---
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list
(email@hidden<mailto:email@hidden>)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden<mailto:email@hidden>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list
(email@hidden<mailto:email@hidden>)
Help/Unsubscribe/Update your Subscription:
This email sent to
email@hidden<mailto:email@hidden>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden