8 Responses

1. Topaz Labs does it again! New product coming - RetouchPRO

   August 2nd, 2009 at 21:16 1

   [...] Re: Topaz Labs does it again! New product coming Wow, that’s what i call coincidence. It happens i’m developing a very similar tool based on the WLS operator: it performs a pyramidal decomposition and let’s you adjust the details at various scales, for more information here. [...]

2. Mike Bedford

   August 3rd, 2009 at 2:03 2

   Manuel, very interesting! Please let me know when you have a review copy ready and I’ll take a look.

3. Manuel

   August 3rd, 2009 at 9:14 3

   Thank you Mike, i’ll let you know for sure!

4. Davide Barranca

   September 16th, 2009 at 21:04 4

   Hi Manuel,
   nice to see that your developing has gone that far!
   I’m looking forward to read more about it – please let me know if you’re in need of beta testing on Mac ;-))
   Ciao e buon lavoro,

   Davide

5. algorithm_implementer

   November 21st, 2009 at 20:42 5

   Very interesting work.

   But how did you implement the solver for the large sparse linear system of equations

   (I+L)*u=g

   that makes the very heart of the algorithm.

   Unfortunately, the authors of the paper don’t give much details about this interesting and by far most difficult to implement aspect of the algorithmt.

   Efficient solvers have been described for similar PDE/poisson/laplace type of equations (Gradient domain HDR compression, Poisson inpainting technique etc.).

   However in the case of Fattal’s WLS algorithm we have an inhomogeneous poisson equation (i.e. the matrix coefficients are not fixed but dependent on the pixel location) which is much mor difficult to devise
   a fast AND reliable solver for.
   Most approaches are heuristic nature and/or come with many knobs&parameters (both of which is especially undesirable in commercial applications)

   The only hint that Fattal et. al give in the paper is a short remark that they use a similar solver than Lischinski et. al in their paper [1].
   However, that solver seems to rely on a preconditioned conjugate gradient scheme, i.e. a classical single-grid iterative solver whose run time may be barely sufficient for “100×600ish” kind of image resolutions but seems infeasible for true multi-megapixel images where a true multigrid scheme would be necessary. However, the literature about multigrid methods for inhomogeneuos poisson equations (contrary to the homogeneous case) is very scant and it does not seem that there exists a solver that is

   1. fast enough for the kind of multi-megapixel multi-scale responsive feedback
   2. reliable, robust and parameter free

   So it would be great if you could share some details about the implementation approach that you have found to this problem.

   ——————–
   [1] Dani Lischinski “Interactive Local Adjustment of Tonal Values”

6. Manuel

   November 22nd, 2009 at 16:21 6

   Hi,

   thank you for your interest and for writing me: indeed, finding a fast AND reliable solver has been a lot of work since that paper, as you noticed, just remarks something about a “Fast approximate solution” [1], however, i remember that computing a Cholesky factorization and using that as a preconditioner it was taking more processing time and memory resources than a slightly modified version of a conjugate gradient solver.
   I rewrote the CG solver on the lines of OpenTissue’s one in order to be able to 1) parallelize as much computions as possible on multiple cores (i’m still working on it) and 2) to maximize vector/sparse_matrix elements access efficiency, rewriting the “axpy_prod” Boost::ublas [2] helper to address non-zero diagonal elements via an ad-hoc lookup table exploiting the symmetric nature of A, saving a whole lot of memory.

   Another interesting “experimental” optimization trick i found out while experimenting, but that has to be used carefully since it directly affect the resulting WLS maps, is built around the fact we need to generate three maps, from the most to the least detailed: the constraint here is to compute the maps in a least-to-most coarse direction, that is, the first map you compute is the “fine”, then the “mid” and then the “coarse”: once the “fine” has been computed we can use the resulting map as a “preconditioning vector” for the subsequent map by converting the “fine” matrix to a vector and using it as the “x” vector for the subsequent “mid” calculation; then repeat the process by using the resulting “mid” wls map as the “preconditioning vector” for the subsequent “coarse” map.
   Since this approach accumulate some error terms, i modified the exit condition of the CG to be bound to the error term itself, so that the solver has no more n-iterations to perform but instead has to reach a decent error threshold before exiting: this ensure every generated map has no more than a well-defined error threshold not differing much from other maps.
   Although this technique works well in most cases, it still can produce some artifacts if error threshold values are taken to the extreme or the previous maps have been generated with a larger error threshold: usually, computing the “fine” map with a target error threshold of 5e-02 and the subsequent maps with a target error threshold of 1e-02 gives good results without sacrifying much of the details.
   The pros: faster on single/dual core processors with as much as 1/1.5gig ram available (4 mins on a p4 2.2ghz for a 21Mpixels, 28 seconds on the i7, Linux 32 and 64 respectively)
   The cons: maps need to be generated in series, thus no parallel map generation can be performed (but detecting a powerful machine isn’t difficult so switching between these methods at runtime can be done and proposed to the user).

   The last optimization trick that can speed up things *a lot* is to compute the WLS on a scaled resolution input image, and use that concept coupled with a UI slider (ie, “Overall quality”): the idea is to have a min/max going from 50-to-100% of the original resolution and use the scaled resolution image for computing the WLS; the important thing here is to always “linearly interpolate” to compute the scaled resolution image otherwise interpolation artifact (avg, grid, bicubic) may appear strongly in the final tonemapped result (*not* in the wls map).

   As a sidenote, i think Intel Math Kernel libraries [4] could be able to solve the linear system (eq. 5) [3] without having to resort to the tricks i’ve mentioned, as well as saturating all cores and minimizing the overhead: however i’ve really never tried the MKL so you may want to give it a try.
   My goal wasn’t at all to compete with their or other existing implementations, but instead just to have a look to the beauty of computational photography, it’s intricacies and it’s implications in the photography retouching process.

   Hope it helps,
   Regards,
   Manuel

   ————
   [1] Dani Lischinski et. al, “Interactive Local Adjustment of Tonal Values”
   [2] Boost’s Basic Linear Algebra can be found at http://www.boost.org/doc/libs/1_41_0/libs/numeric/ublas/doc/index.htm
   [3] Dani Lischinski et. al, “Edge-Preserving Decompositions for Multi-Scale Tone and Detail Manipulation”
   [4] Intel Math Kernel libraries can be found at http://software.intel.com/en-us/intel-mkl/

7. algorithm_implementer

   November 22nd, 2009 at 20:10 7

   Hi Manuel,

   thanks for the detailed and open sharing of your ideas. This is very well appreciated !

   I too have been in the process of developing something similar in the context of an image segmentation engine which essentially boils down to the same system of equations to solve.
   Your results for 21 Megapixels:

   > 28 seconds on the i7

   are impressive (I assume that this is the time needed to compute all three maps to an solver accuracy of 1e-02).

   How much of smoothing (lamnda parameter) do you use for the “coarsest” grained level ?
   The main problem seems to be the fact that processing time increases not only with image resolution alone but also (and even more rapidly) with increasing smoothness factor.

   However, in order to achive a complete scale space decomposition, it is desirable to have a very strongly smoothed version at the coarsest level (i.e. nearly as smooth as the completely smoothed, i.e. constant image )
   In my experience the processing time roughly doubles with a doubling of the smoothing scale …

   The typical approach to solve such problems with the slowly converging low-frequency errors would be to use some multigrid variant of the linear solver.
   However, after having experiemented a lot with multigrid solvers (adapting the restriction/prolongation operators, algebraic multigrid etc.) there seems to be no significant speed up compared with a well tuned single-grid conjugate gradient solver….

   So did you too experiment with the multigrid approach and if so, how did the results turn out ?

8. Manuel

   November 22nd, 2009 at 20:59 8

   I just tested it now and it take 38 seconds on the i7 to compute all three maps on a 50%-scaled input, making use of the above optimizations, thus with an accuracy of 5e-02 on the first map and 1e-02 on the subsequent two maps.
   Keep in mind that i’m not able at this time to push all the 4+4 cores at work, it stays at 50/60%, so there is still room for improvement: too bad i can’t remember the exact settings for having the i7 do the same results in 28s, but i remeber i got it by adjusting lambda/alpha coefficients, as you already noticed the processing time roughly doubles as doubling the smoothing scale.
   The factory settings of my application are Sm(oothing)/Sh(arpness)/Err(or) this way:

   fine map: 0.1|1.2|5e-02
   mid map: 0.4|1.4|1e-02
   coarse map: 0.5|1.8|1e-02

   I also noticed the fine/mid map params are good to be kept the same for most of the images i tried, the coarse params, instead, it seems a lot of different and good-looking variations can be achieved just manipulating its lambda/alpha between the nominal 0.5|1.8 and 0.7|1.7; the time spent on the coarse map will vary a lot between choosing a lambda=0.5|0.7.

   I never tried the multigrid approach and unfortunately i don’t think i’ll have the time to work it out soon but given the genericity of the math engine you are working on, you may be very well interested into “edge-avoiding wavelets”, that some sort of second-generation wavelets.
   I recall that recently there was a work in the MRA field that depicted how WLS filtering can be done and implemented with wavelets, basically it can be achieved multiscale edge-preserving filtering at computation times which are linear to the number of pixels, avoiding the need for solving large systems of equations: i’ve still not studied the paper [1] yet (there is a lot of stuff going on here!) but Fattal himself states and demonstrates it’s robust enough.

   ————
   [1] Raanan Fattal, “Edge-Avoiding Wavelets and their Applications” (http://www.cs.huji.ac.il/~raananf/projects/eaw/index.html)